atsameip.com was basically an experiment. i saw some other similar site and i thought "i can think of a way to make a site like that" so i started building my
clone.
you might think that this is like stealing an idea, but even if i did get the idea from another site i could not know how it worked inside. so i had to come up with my own way of producing the wanted result and thats basically the fun part
.
so my first idea on how to make this was to make a table with 1 row for each IP and each row would have a field that contains a textual list of domains associated to this ip.
even to make sure the data doesnt get too big instead of just 1 "ips" table i made 1 for each first digit of an IP (from 1 to 255) like this "ips1", "ips2", "ips3" .... "ips255"
and well that allowed me to store as much data in as little space as possible.
and the basic idea is whenever someone searches for a domain, you ping that domain and add it to the list, so as more ppl come to checkout their domain the database grows.
but in practice after i just started this wasnt very good cos the site beeing unknown and no one visiting it, the database was not growing much and the results were poor compared to the *competitor sites
so i decided to build some scripts that would import lots of domains in my database.
the first primitive experiments included a script that generated random domain names and just tried them all and added the ones that pigned (think xo23hjd.com) and a crazed web crawler that was visiting random urls and saved all found urls on the pages and then visited them and went on, pinging all the domains it encountered which ended up completely hosing my server.
but finaly i started using a huge list of domains (only seems to have .com domains but lots of them) provided online by some site that i would rather not name (at least until im finished crawling their list...)
anyways so then i started having a more interesting amount of results but i was having 2 problems:
first with the quantity of domains getting large (a few million, on a few hundred thousand IPs) i had a problem that was caused by the fact that some of the IPs had very large amount of domains pointing to them. that made the rows of these IPs be heavy to read since they contained a long "text" content, and sadly since the whole list of domains was in 1 text, even when you wanted only to show a few of the domains of an IP you still had to read the whole list from the DB and parse it. so the result was that atsameip was generally fast, but as soon as one of the IPs with large amount of domains pointing to it was part of the results, it made the query quite long. thats a problem that i wanted to find an intelligent way to fix.
another problem that i had was that importing (and testing) all the domains from that huge online list of domains i found was taking a very long time (there might have been at least a few hundred thousand and maybe even million pages to crawl and each crawl was taking many hours) and the process was also using a lot of CPU on my DB (i have other sites on that server...)
but for the first version of the site thats how it was... i eventually stopped crawling the huge list of domains, having only crawled the first pages (now i have a lot of domains that start with 000000 like 00000000123sex.com ect...) and i was not thinking much of the site, using it once in a while...
time has passed and i have been creating various other sites and continuously improving my programming skills. and someday i noticed that i was starting to get some traffic on atsameip.com. so i went back to look at the site and asked myself "how can i improve it with my new skills?"
so first i wanted to restart the script to crawl the huge list of domains and get them all-in once and for all; thinking that maybe if i re-orchestrated the mysql queries differently i might be able to make it work faster.
when i started to try to write a more efficient script i felt that it was hard to build something efficient with the way my information was currently stored (255 separate tables with 1 row per IP and all the domains bundled in 1 text for each IP). so after spending a while trying to work something efficient on the old system, i started becoming crazy about redoing a brand new system, a more efficient one: my idea was to create a single table where each domain would have a row containing its name and its IP. of course that would mean 1 huge table, with a looots of rows... and it would ultimately use more space on the server because some IP address would be repeated. but that would allow me to manage the data a lot more easily, like easily verify if a domain is already listed or not and really select only 10 domains from a certain IP even if there are thousands. I felt that the new system might possibly fix my 2 problems, the problem where IPs with lots of domains slow the thing and the problem where crawling the list of domains is very slow.
So i started building the new system.
first i created that new "domains" table that would contain all the domains and then i wrote a script that would import all the domains from the old tables into the new table.
and then i found out that importing this data was taking forever! as im writing this i cannot be exactly sure but i suspect that the old tables might have contained about 2 or 3 million domains and now after many days i have managed to transfer about half of it in the new table!! lol im still continuing though until i will have transfered it all of course, i also improved the script many times since its first version now i guess its about as fast as i can make it for the moment.
then i made a new version of the script to crawl that huge list of domains to work with the new all-in-one table and since the new table makes it much easier to verify if a domain exists, i was able to reduce ridiculously the time it takes to crawl and test the domains of the list. lets just say that at the speed it was going before it would have tied up my database for about 30 years before it was done reading the whole list, but now i estimate that it will take about 3 days.
and then i also make an entire new version of the homepage to work with the new system and since i was waiting for this super long data transfer to be finished i started adding lots of options, like country flags next to ip, whois information and a forum and a contact us page
so you might think that all is well in wonderland, only need to wait for the data transfer to be finished and kick in the new homepage and bing we have a new super update of the atsameip system.
but sadly the new system, even if it was working fantastic, had a little problem: it was veeery slow!!
while the old system was generally lets say 9/10 fast and once in a while 2/10 fast, lets say the new system was steadily 4/10 fast... not good at all...
of course while im still importing data and have all these crawlers/table transfering scripts running, it makes it even slower. but i tested whit all these temporary scripts not running and while it was a little bit faster it was still slow enough to be annoying and generally not nearly as fast as the old system....
so at the moment the old system is still on the homepage, looking quite much exactly and working exactly like it always did. but theres a link on the home page to try the new version of atsameip which im still working on.
it has lots of new features that im proud of, including country flags and full support for internationalized domain names. but im still working on some ways of increasing the speed of this thing before i release it on the first page.
and i must say that even if it was not easy, i did manage to imagine a way to drastically speed up the new system but it involves pre-organizing a massive amount of information which i have a script working on at the moment and i estimate it will take several hours, so that gives me some time to write this long story
but basically, my point is that if all goes well, probably in a couple of hours, days max, the new system shall be ready, full of new options and Fast! ill keep you posted on the development.
Welcome to AtSameIP 2.0
replies:
