WedTM's Blog

Web developer, father, husband, and awesome dude. All rolled into one.

Twitter's Real-Time URL Fetcher Using Cassandra and Memcached

nosql:

Twitter’s real-time URL fetcher, code named SpiderDuck, is an excellent example of how NoSQL databases fit in the architecture of today’s systems:

Metadata Store: This is a Cassandra-based distributed hash table that stores page metadata and resolution information keyed by URL, as well as fetch status for every URL recently encountered by the system. This store serves clients across Twitter that need real-time access to URL metadata.

SpiderDuck is also using memcached:

Memcached: This is a distributed cache used by the fetchers to temporarily store robots.txt files.

SpiderDuck Architecture Cassandra Memcached

Original title and link: Twitter’s Real-Time URL Fetcher Using Cassandra and Memcached (NoSQL database©myNoSQL)

Yogscast and Markus: My Two Cents

First and foremost, I know both parties involved, and think they are wonderful people. That being said, we all make mistakes, and it shouldn’t ever mean we should stop supporting or boycott a brand/person because of it. We’re all human here.

The issue here lies in the fact that NOBODY has ever done what’s being done here. Very few games ever try the release cycle and transparency that Markus & Co. have. Even fewer (if any?) succeed. 

Yogscast are spoiled diva’s. Duh. They should be, they are amazing! Have you ever seen their shit? It’s awesome! That’s what the fans make them, YOU love the fact that Simon and Lewis are ass hats. I do too.

So you mix a very tired Mojang crew, with the weight of high expectations for an event that is really just an over-glorified game launch that had to be stretched out over 2 days, and never doing ANYTHING like this ever, and some spoiled divas (even if they are justified) and obviously ego’s are going to be stepped on.

I think this points to an even bigger problem. A while back, Yogscast made a comment about us allowing adf.ly links for mods. We never issued a response because, after all, they have every right to be pissed about that. Regardless of the fact that I personally thought it was silly, we never made an official statement.

The people to blame are the people who take it too far. The people who take a single off hand comment, and twist it and turn it around so many times, that the original meaning is lost and all that’s left is a twisted web of lies.

I urge you, the fans, to take to heart what you really love about these people, and realize that they are just normal guys. They drink, they like to play video games, they laugh and cry, they get pissed and say things they don’t mean. After all they’ve done for us, do they really deserve us taking an off-the-cuff comment and make it a huge deal? I mean FFS, Minecraft was just released! I’ve been personally working and waiting for this moment for two years. That is what we should be focusing on.

As for me, I love everyone. In fact, this was my view from the Nether part at MineCon:

This Guy Really Likes MCSL

So I was checking through some analytics and found this youtube video popping up. I decided to see what it was, and much to my surprise, someone has made a video on how to post a server on MCServerlist.net. I guess I should rework some things if it requires a tutorial!

http://www.youtube.com/watch?v=XnGTRw69BmU

Voting on MCSL

Hey, guess what? You can now vote on servers at MCSL! It’s a feature that’s been requested alot, and I think it will give people the chance to promote their server in a much larger way.

I haven’t quite figured out how I’m going to add the votes into the ranking, but I will let you guys know, and I’m open to suggestions. Just pop in #mcsl on irc.esper.net.

As always, let me know via the Feedback tab if you find errors or anything.

How the ranking system works on MCSL

I’ve gotten a lot of questions on how ranking works on MCSL, and I’ve been trying to find the best way of showing people. The best way that I’ve found is to simply post the source of the ranking algorithm. 

So here it is: https://gist.github.com/1024062

I hope this squashes some people’s questions on the ranking algorithm.

New Changes on MCSL

I’ve just finished pushing the new code to the production boxes for MCSL. I made a bit of a concession on the new “New Servers” tab on the front page. It shows any servers created in the last 7 days. 

Pagination is where the real issue lies. The gem I’m using for it (Kaminari) has a weird issue when trying to paginate multiple models on the same view. It can’t keep the variables seperated, and ends up adding both pagination linkst to the URL, so it’s a bit clunky atm.

After I get that fixed, it should run much cleaner.

I’ve also added a new button in the dashboard that will show you a history of EVERY check made on your server, and what it’s result was. This should help people who are having trouble tracking down why their server is going down on the list.


I’ve also added a link to the blog on the front page, so people can start realizing that this is where I do most of the update chatter.

Fixed Page Views On MCSL

I’ve fixed the first round of errors that I’ve found, and MCSL seems to be functioning reasonably well.

I’m also in the process of adding a “New Servers” section, that’s a tab on the front page, I’m having a slight issue with my pagination though, so right now, it’s not on the production site.

As soon as I get that fixed I’ll push that live.

MCServerlist Update

Ugh, people really irk me sometimes. 

For those of you who don’t know, I just had surgery on my T11 and T12 vertebra, and have had to suspend work on the new MCSL for a couple days. I’ve gotten several emails from users of MCSL asking for help with their servers, as the current site is COMPLETELY broken. 

I designed the current MCSL to handle hundreds of servers. I never thought there would end up being THOUSANDS of servers, and the site was never optimized that way.

That’s where the beta comes in. To show people that we were working on the issue, we started letting people come in and see the new site before it went live.

We’ve had tons of people donate, and this helps keep the $1,000+/month server bills down, but we have also had several people of late either issue PayPal complaints, or (this is my favorite) threaten to take us to the BBB. Well, first off, the BBB isn’t going to do anything, since we’re not a member organization. Secondly, you’ve DONATED, not purchased anything. We love our donators, and even have some special badges that have been made, just for them. But if you are expecting something in return, please don’t donate. We don’t want to deal with any of the drama it brings.

On the beta, after I get feeling a bit better, I’ll be putting the finishing touches on the site that will get it to it’s CURRENT stage on production, except it will be stable.

Then I will release it. We’re talking 2 weeks, tops.

After that, I’ll start on the other features that I had planned and will start getting those added in as well.

To those of you who don’t act childish, and understand that there’s really only one developer, I appreciate you.

- WedTM

 WedTM - By TheDr.

 WedTM - By TheDr.

MCServerlist.net Needs Your Help!

MCSL has been growing exponentially over the past few weeks. I’ve been feverishly working on a complete redesign. One that is both more stable, and has more features.

I’ve also purchased two brand new, 16 core, 12GB DDR3 RAM, 1TB Raid 10, servers. These are uber fast. The site should be moved over to these in the next few days, however, they are starting to tap my budget dry.

For those of you who don’t know, Minecraft is how I make my living. With the work I do with Curse on the Minecraft Forums and Wiki, as well as MCServerlist.net, that is what supports my family!

Our advertisers (Especially Multiplay!) have been amazing thus far, but we’re in need of a bit of capital to bring the site to it’s fullest potential. This is why I’m putting up a donate link. I want people to be able to support MCServerlist, and I think this is the best way of doing it.

I’ll be giving people limited access to the new MCSL v2 over the next few weeks, however, if you want in sooner, I’ll be giving people who donate over $10 exclusive access to the beta as it’s being developed.

Also, any server who donates over $30 will have a flag set on their server which will show your server name in bold, and also weight your server (ever so slightly) so you appear higher in the rankings. You’ll also get a cool supporter badge on your servers page.

This weighting won’t be drastic, so if you’re at 50% don’t expect to jump to the front page, however, if you ARE 100%, you will be at the top of the list, above other 100% servers.

If your interested in donating, please click the donate button located here:

http://mcserverlist.net/donate