Hacker Read

q7xvh97o2pDhNrh · 2023-11-19 20:49:59

That's fascinating that the total size is so tiny — only 2.4 TB?

I assume this must be only the text portion, and heavily compressed?

reply

andai | karma 8456 | avg karma 2.45 · | 2019-06-25 17:17:48+00:00

It's mostly text, isn't it? So more like 10kb.

mathnode | karma 870 | avg karma 2.99 · | 2014-03-03 14:04:10+00:00

That's nearly 4.5TB if it was gzipped plain-text.

alchemist1e9 | karma 1206 | avg karma 1.47 · | 2023-04-15 20:33:38

Am I the only one who is surprised how small the storage size is? I understand it’s a lot of text but to realize it easily fits on a few cheap HDDs raided together is still incredible to me.

ars | karma 27176 | avg karma 2.04 · | 2022-07-03 16:05:07

It's 7TB compressed. If it's text you'd need about 70TB to decompress it. It's probably mostly images though, so probably not quite that bad.

Houshalter | karma 10632 | avg karma 2.14 · | 2013-08-20 23:56:38+00:00

Almost 9 GB, though it's significantly smaller if you download the version without images (which my Wikipedia reader can't render anyways.) IIRC pure text is only like 2 GB.

There is a torrent for them somewhere.

As large as that is, I am surprised that it wasn't huger than that.

reply

pradn | karma 4976 | avg karma 4.37 · | 2020-07-19 16:05:05+00:00

The English Wikipedia text (no images) is about ~20 GB compressed, but ~60 GB uncompressed.

jck | karma 402 | avg karma 3.02 · | 2023-11-16 10:25:58

My guess is not by much. Text doesn't require a lot of storage. After looking at a few random files my estimate is that the size of the full compressed dump would be about 30x (~10 GB)

calibas | karma 3473 | avg karma 4.48 · | 2022-08-25 00:32:01

It's only 8kB compressed.

asutekku | karma 1384 | avg karma 2.69 · | 2020-10-25 22:20:24+00:00

36gb of compressed text is a lot though.

ocdtrekkie | karma 24767 | avg karma 2.6 · | 2017-08-03 17:23:39

It's 11.9 GB of text (5.3 GB zipped). So smaller than quite a few video game downloads.

antimatter15 | karma 4727 | avg karma 16.36 · | 2011-12-31 01:09:47+00:00

It's most of the large articles, the entirety (stripped of citations and image metadata) and compressed is more like 4GB. For comparison, everything (including citations) as a wikimedia XML dump is about 7GB bzipped and thirty something uncompressed

Groxx | karma 17784 | avg karma 2.5 · | 2010-12-03 18:19:28+00:00

size: 8.50mb

I doubt that's all of it.

reply

SleekEagle | karma 947 | avg karma 4.6 · | 2023-09-06 09:15:49

Fun fact: All of Wikipedia (text only, compressed) is like 20 GB

aswanson | karma 5136 | avg karma 1.92 · | 2007-08-30 00:47:59+00:00

Only 4.4 gb as uncompressed text.

michaelmior | karma 5938 | avg karma 1.89 · | 2017-08-10 16:06:57+00:00

No, it's only ~500KB.

vezycash | karma 10886 | avg karma 5.64 · | 2020-05-28 18:44:17+00:00

According to the about page, it's 65KB.

Skunkleton | karma 3307 | avg karma 2.48 · | 2019-07-03 18:43:02

The article actually says that only ~1 TB was downloaded by the researcher. The real size is unknown.

hajile | karma 6470 | avg karma 2.37 · | 2021-03-26 20:25:15+00:00

I'd point out that it's 1.2MB uncompressed, but only around 390k compressed.

jussij | karma 2569 | avg karma 2.45 · | 2014-05-07 00:06:17+00:00

At 2MB that is hardly a large file.