Some stats on the words used on this site

[estadiare]

I am currently working on an anonymized archive for this site that I can host on tor in case this site ever goes down. As i need a bit of distraction, I've decided to make the search index and the subsequent search engine on my own from scratch :)

i realized that some of my code could also be used to figure out some interesting stats

on this site, there are 900'000 unique words, of which 270'000 are relevant

some examples of the non-relevant words:

- fibroymalgia
- politcial & policitally
- moneypox
- therminologys
- alphafold

even though most recorded words are irrelevant, 95%+ of all total words written on this site are in the relevant category.

most of the "irrelevant" words are typos, scientific terms or terms only used in a certain field, usernames and non-english terms

the most popular word around is "to", with at least one mention in almost any single thread

The longest word ever used is
mjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoieher

as found in this thread:

https://sanctioned-suicide.net/threads/the-shit-i-have-to-deal-with-on-reddit.45352/

 

[Ginnn]

Now "mjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoieher" is my favorite word

 

[makethepainstop]

Pronouncing it and learning to spell it correctly, are the words only drawbacks.

 

[Angst Filled Fuck Up]

Now "mjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoiehermjvvianeslsslsfnfnccqpqwodqmdamamcamsfoihfewofhwifgwfhueawprhoierwgievofdnvoieher" is my favorite word

That's a town in Wales, isn't it?

 

[chronicallybroken]

I love this! I find this stuff really interesting. Not the technical side though because I don't understand it

 

[locked*n*loaded]

Why are fibroymalgia and monkeypox irrelevant? If someone suffers from those conditions and it causes them physical pain, and consequentially depression, resulting in them wanting to CTB, how can they be irrelevant?

 

[estadiare]

Why are fibroymalgia and monkeypox irrelevant? If someone suffers from those conditions and it causes them physical pain, and consequentially depression, resulting in them wanting to CTB, how can they be irrelevant?

these are typos, it says moneypox and fibroymalgia

 

[locked*n*loaded]

these are typos, it says moneypox and fibroymalgia

Sorry about that. I guess I'm seeing things that aint there. Don't know how I missed that. Well then, I agree, those words are irrelevant. Carry on.

 

[artificial_ineptness]

This is pretty cool, I've always spent an inadequate amount of time nerding over data

Do you plan to do more analysis / share some more stats? The first thing that comes to my mind with data like this is to see over- and under- represented words (This Kaggle dataset might be an okay start to be compared with?).

Also, how did you filter out the "irrelevant" words? Maybe you used some dataset of common English words (or maybe just got rid of rare words?)

 

[estadiare]

This is pretty cool, I've always spent an inadequate amount of time nerding over data

Do you plan to do more analysis / share some more stats? The first thing that comes to my mind with data like this is to see over- and under- represented words (This Kaggle dataset might be an okay start to be compared with?).

thank you! Unfortunately i don't plan to do more data analysis for the moment

Also, how did you filter out the "irrelevant" words? Maybe you used some dataset of common English words (or maybe just got rid of rare words?)

for the search engine, I have a list of words and within one word entry there is a list of threads and an associated score. For example, the entry for 'dataset' looks like this (word|thread1:score1,thread2:score2,...):

dataset|11877:4,28438:4,29980:7,30211:2,30739:8,34679:4,43044:0,7608:2,77788:4,82081:29,85404:2,86362:6

so for example in the thread https://sanctioned-suicide.net/threads/n-failure-from-final-exit-question.85404/ the term 'dataset' is only mentioned in one comment so it only has a score of 2

the thread https://sanctioned-suicide.net/threads/status-of-mental-health-2022.82081/ on the other hand has several mentions in the OP post, so it gets a score of 29


my technique for filtering out 'irrelevant' words at the moment is to sum up the scores for a word and if it doesn't reach a certain threshold it is filtered out