Hacker Read

mfisher87 · 2016-11-30 23:36:45

Thanks! It wouldn't be so bad if there weren't so many duplicates :)

okal | karma 2002 | avg karma 7.12 · | 2012-02-14 23:21:21+00:00

Thanks :) Duplicate entries are not allowed, but I'm still trying to figure out the best way to handle them. If you come across one, simply flag the last to be entered so I can manually delete them. SICP also got added a couple of times.

bd_at_rivenhill | karma 591 | avg karma 2.39 · | 2013-04-02 13:42:26+00:00

I'm seeing a fair few duplicates in the results, probably need to work on the algorithm for filtering these out.

vlunkr | karma 5913 | avg karma 3.28 · | 2016-02-05 12:24:16

This is great! My only complaint is that it comes up with lots of near duplicates. The images look they are different frames, but the quotes they reference are the same

minimaxir | karma 67739 | avg karma 7.48 · | 2015-01-16 23:29:13

You have a lot of duplicate entries there.

freddy | karma 50 | avg karma 0.88 · | 2009-03-19 15:20:10+00:00

yeah I agree, that is a good idea to help clean up duplication. Thanks for the comments, let me know what else you can think of.

clone1018 | karma 794 | avg karma 4.49 · | 2011-12-31 14:46:04+00:00

It shouldn't allow duplicates, I'll work on the rest.

michaelmior | karma 5938 | avg karma 1.89 · | 2021-04-20 23:50:58+00:00

Looks very cool! A couple things however that I noticed. There seem to be a lot of duplicates which clutter the search results. Also, the first thing I looked for was available, but the correct result didn't appear until the second page.

conradludgate | karma 1079 | avg karma 2.55 · | 2023-05-06 07:11:51

During search, we do remove duplicates. It's not a bad idea though and I'll see how we can support it

michaelhart | karma 339 | avg karma 3.39 · | 2010-11-09 18:41:50+00:00

I just modified the way the queries run; duplicates should be MUCH less common now.

leoh | karma 2942 | avg karma 2.93 · | 2017-02-24 03:50:02+00:00

Looks cool, you have a lot of duplicates, though.

   $ cat sorted_unique_cf.txt | wc -l
    7385121

   $ cat sorted_unique_cf.txt | uniq |wc -l
    4287625

fishtoaster | karma 5446 | avg karma 10.39 · | 2012-02-27 16:54:46+00:00

Yeah, I agree. I didn't expect such a quick influx of new bits- definitely need a way to detect duplicates.

Jonovono | karma 999 | avg karma 1.73 · | 2022-10-15 17:38:18

Nice! haha thanks for checking it out! I noticed an error in my Duplication template, so I think I was creating too many per user. You are most likely around first 5 users ;p

nikisweeting | karma 2396 | avg karma 2.46 · | 2017-02-24 04:02:36+00:00

I had duplicates, it's actually only 4,287,625 (still a lot though).

Fixed the duplicates: https://github.com/pirate/sites-using-cloudflare/raw/master/...

reply

bovermyer | karma 7932 | avg karma 3.1 · | 2015-03-18 13:29:06+00:00

Very interesting concept. I signed up; we'll see where it goes.

One thing to note, though - on the left I'm seeing a number of duplicated entries, with the duplicates immediately after the original. I'm using Firefox, if that matters.

reply

gpm | karma 17771 | avg karma 3.89 · | 2022-08-30 17:40:59

Huh, there's a ton of duplicates in the data set... I would have expected that it would be worthwhile to remove those. Maybe multiple descriptions of the same thing helps, but some of the duplicates have duplicated descriptions as well. Maybe deduplication happens after this step?

http://laion-aesthetic.datasette.io/laion-aesthetic-6pls/ima...

reply

siong1987 | karma 2169 | avg karma 5.12 · | 2008-12-21 23:08:10+00:00

I just pruned all the duplicates right now. Will work on the today, last week, last month sections tonight.

staunch | karma 28228 | avg karma 4.34 · | 2011-09-24 02:04:51+00:00

We're still testing this out with real data, but it looks like it's actually quite useful to have duplicates in the database. The key is how to return results to someone coming along later. We're working on that now.

We committed upfront to not letting the site become overrun with "Yahoo Answers" style duplicated/low quality stuff. We'd much rather delete useless stuff than get an extra page view or two.

reply

0x7f800000 | karma 138 | avg karma 4.6 · | 2018-02-22 14:25:31

It would be nice if 5x duplicates didn't show up in the RSS feed.

robbles | karma 703 | avg karma 2.58 · | 2012-02-27 02:43:53

Awesome concept! Thanks for building this.

One possible issue - There needs to be some kind of search for finding similar "VimBits". I wanted to add some of my favorites, but I have no way of knowing whether they're there already without reading through everything on the site.

You'll probably get a massive amount of duplication unless you can implement some kind of "suggested duplicates" on the create page, like Stack Overflow.

reply