Hacker Read

jwcrux · 2014-08-11 16:59:10+00:00

Why not use PRAW? It's very mature, useful library using the Reddit API.

onion2k | karma 46633 | avg karma 4.72 · | 2020-10-26 17:07:53+00:00

Reddit has a pretty decent API. PRAW is the most commonly used library for it (in Python), but there's https://github.com/not-an-aardvark/snoowrap if you're set on JS too.

habitual_coder | karma 3 | avg karma 3.0 · | 2016-11-15 17:38:43+00:00

read the api documentation for reddit, follow the rules, use python+praw to make requests. Writing a script with BeautifulSoup or similar seems like a headache.

seesawtron | karma 3723 | avg karma 5.36 · | 2020-06-18 18:33:55+00:00

Awesome. Didn't know about reddit API wrapper praw! Thanks a ton, will give it a try.

One question: How does the locally run python script feedback the scraped data to bot user account app?

reply

Arimbr | karma 315 | avg karma 2.94 · | 2020-04-09 16:17:40+00:00

Right! I used the official Reddit API. I created an APP, got the credentials for the API. Then used the Python library PRAW to consume the API. https://praw.readthedocs.io/en/latest/

It took me 36 hours to collect the 4M posts. Reddit API returns results in batches of 100 results, and then sleeps for 2 seconds.

You can find some more details on how it was built here https://blog.valohai.com/machine-learning-pipeline-classifyi...

I can publish on Github the repository that runs two commands to collect the data if your are interested.

reply

toyg | karma 35240 | avg karma 2.97 · | 2020-04-09 16:13:49+00:00

Probably with the official Reddit API? There are several libraries for it.

kgc | karma 401 | avg karma 1.21 · | 2014-06-11 08:32:31+00:00

Why not just port to the Reddit codebase? The functionality seems similar.

loktarogar | karma 819 | avg karma 3.25 · | 2020-09-24 07:10:14+00:00

the base reddit api is pretty good - just tack a .json onto any url for an example

avree | karma 2188 | avg karma 4.01 · | 2023-06-20 16:53:40

Reddit's API is free for my use cases. From their documentation: "Our API allows free access to moderators and developers creating these tools for non-commercial use cases."

plastic_bag | karma 68 | avg karma 1.45 · | 2021-10-21 01:45:00

Why not use libredd.it? It's a privacy friendly frontend for Reddit without any JS. You can host it yourself or run it off a docker container.

TazeTSchnitzel | karma 26116 | avg karma 2.81 · | 2012-10-27 00:02:59+00:00

Isn't that the non-public internal reddit API? They have proper APIs for other things, I think.

gitgud | karma 5259 | avg karma 2.23 · | 2018-08-21 03:26:27

This is such a cool idea! Does it scrape reddit, or use the API?

Havoc | karma 16454 | avg karma 2.29 · | 2021-08-12 19:57:32

> Additionally, its trivial to stream and analyze the entirety of Reddit comments and posts in real time (check out PRAW if you like Python).

Chunks of it. Don’t think praw and the api can do all live

reply

dawie | karma 2253 | avg karma 2.25 · | 2007-05-30 15:08:51+00:00

I thought Reddit was rewritten in Rails, but after some research I realized that I am wrong. Its Python.

StavrosK | karma 1 | avg karma 0.0 · | 2011-08-02 07:06:44

I'd use the reddit/HN codebase, or a reddit/HN clone.

karanbhangui | karma 979 | avg karma 2.56 · | 2009-06-13 06:56:57+00:00

Why not just use the reddit codebase?

gkbrk | karma 2202 | avg karma 4.56 · | 2023-12-18 22:04:45

Reddit has an API and many clients.

intothemild | karma 792 | avg karma 3.58 · | 2023-06-08 18:23:53

Not the worst idea.... Build an API that sits as a middleman to reddits api, it's paid for... But caches requests to cut costs.

Possibly use some amount of web scraping too

reply

pixelatedindex | karma 508 | avg karma 2.08 · | 2023-06-13 11:55:01

Reddit's APIs are not user-based from what I can tell.

O_H_E | karma 1534 | avg karma 1.84 · | 2019-01-27 04:30:48+00:00

Reddit have a Public API, thats how all these unofficial clients work. Don't know for sure if this site is using it. (It probably is)