SEPTEMBER 1 To ulTRAX
From: jgreely@corp.webtv.net (J Greely)
Date: Tue, Sep 1, 1998, 10:23am (EDT-3)
To: ulTRAX@webtv.net (Killer<> ulTRÅX <> Willie) Subject: Re: Future of WTV Hidden In URLs?
ulTRAX@webtv.net (Killer<> ulTRÅX <> Willie) writes:
But there are entire other URL groups which exist, yet do not seem to have been put to use.
If they weren't in use, we wouldn't pass them out.
wtv-* wtv-star (this may be "Network Unavailable")
I'll give you this one for free. These two are the same, and are indeed tied up with the various "unavailable" messages.
wtv-epguide
One of my personal favorites.
-j
======================================================
SEPTEMBER 1 To ulTRAX
From: jgreely@corp.webtv.net (J Greely)
Date: Tue, Sep 1, 1998, 7:10pm (EDT-3)
To: ulTRAX@webtv.net
Subject: Re: Future of WTV Hidden In URLs?
Jgreely you are such a tease.
Gosh, I hope so. :-)
LOL I assume that the URL groupings (got a better, more technical sounding name?) are being used somewhere..
They're all in use, and, BTW, we just call them
services.
just not in 1.3/4 or 2.1/2.
The services on the production machines are in use by our customers, all of whom are running one of those builds.
Then there's wtv-log, wtv-notices, and wtv-weather.
Those are actually pretty obvious names.
As for epguide or any other URL, my record for guessing any URL correctly has been zero for zero. So how about a clue on epguide?
Hmmm, let's see; you use it regularly, and you'd be unhappy if it stopped working, although it might take a while for you to realize that it was broken.
Then there's Tricks in which the bizzare names like Willie and JiffyPopORama must have been chosen with security in mind....
More whimsy than anything else.
Well you'll be hapy to know that there have been no reports of anyone getting back into Daily on this end...
It *would* be a neat trick these days, and almost certainly require the help of an insider; many of our developers needed written instructions. I have to make sure a few specific changes don't get undone the next time they change the software on those machines, though.
Most I heard of is someone who says they went back to Production 1.3.
Possible; we're not forcing 1.4 on everyone yet, and 1.3 is still on Willies.
Mattman did actually lose Doom.... whether it's something you did....
who knows. He thinks it's because he tried to update the game page and deleted the game. He's pretty upset with himself.
We were going to do it, but he beat us to it. :-) We may still run a sweep to make sure it's removed from everyone's hard disk. Only a small percentage of the people who were discovered with internal builds are posting to newsgroups.
In the "unintended consequences" dept. the "fix" for stayconnected has given someone a new way to make a bomb.... though you have to dupe them into forwarding the bomber's post.
We originally changed the timeout from 24 hours to a few minutes, but we didn't want to risk having the legitimate users (people giving demos, primarily) think it still worked and get burned. Besides, we really enjoyed the way it behaved when we dropped the timeout to zero. :-) We'll probably reset it to 30 seconds or so soon, if they don't get the new tricks server finished first. That or we'll create a new page that requires a separate password, so that demos can be run against production more easily.
So when we go direct to an WTV IP (http://IP:port) and get bounced backed to the reglogin screen.... technically what's happening?
Most of our services won't speak to anything that's not one of our boxes, or to a box that hasn't logged in. A direct connection doesn't look right, so the service sends back a "go log in" message, and the box interprets it. Many of the posted "how to get an upgrade" tricks are nothing more than things that trigger a new login, and if the upgrade is available when the login happens, they get the offer. Of course, I won't explain *why* the upgrade isn't being offered continuously, and I can't explain why some don't get it at all.
-j
======================================================
SEPTEMBER 2 To Mattman
From:J Greely
Subject: Re: HI!! To: MattMan69@webtv.net (Matt Man) Date:Wed, Sep 2, 1998, 5:04pm (EDT-3)
Just wanted to know if you would tell me when games will be available for the plus?
Release dates are obviously something we don't discuss. In this case, I haven't even looked at the schedule recently, and I'm sure it will slip from the date I last saw. If ever!
as you most likely know i had the games for a short time and they seemed to work fine, why not let us have them now?
There are licensing issues, among other things, and there actually are some bugs still being investigated. What we don't want to do is force a customer to download something as large as Doom more than once, especially the poor souls who don't have local access numbers.
Heres a few more questions. what are daily/testdrive for?
Testing new stuff. "Daily" refers to how often we change the service software on it. There's usually at least a few things broken, and the machines it runs on don't have the capacity to handle large numbers of users.
is it possible to get a legitimate account there?
Not unless you work for us, and even then, most people don't use it. It's not particularly stable or fast for actual web use.
Also can you get me a job at webtv?
I'm not a hiring manager. A resume sent to me would just get forwarded to HR, so it's quicker to send it there directly (there's a list of current openings somewhere on http://webtv.net/).
or even make me a beta tester?
Becoming a Previewer is certainly possible, although I don't know any of the people over there well enough to say whether or not they'd want you. At the moment, I think you'd need to wait for the heat to die down; everyone who uses the internal/Daily builds has been inconvenienced by the changes we had to make to Tricks, Willies, and Daily itself, so anyone associated with that is not on our list of favorite people right now. Some people were basically unable to work for several days, and they still have to go through several extra steps to do their jobs.
I really enjoy the challenge of finding out as much as i can about your network and how it works.
I understand the impulse, and would under other circumstances be sympathetic. The problem is that a lot of people only want to make malicious use of that information, and even if we can tell the good guys from the bad guys, anything the good guys say in public helps the bad guys. For instance, posting a message with bright red text announcing that Stayin' Alive is now a bomb, making sure that people who weren't clever enough to think of it themselves would immediately start using it all over the place.
-j
======================================================
SEPTEMBER 2 To ECWFRK
From: jgreely@corp.webtv.net (J Greely)
Date: Wed, Sep 2, 1998, 4:50pm (EDT-3)
To: ECWFRK@webtv.net (F T W)
Subject: Re: TO WEBTV Pt.2
ECWFRK@webtv.net (F T W) writes: Why, when the stayin' alive trick was discovered to be out, was it turned into a bomb?
Short term solution, so that the few legitimate users (people running demos, mostly) would immediately know that it no longer worked. It's a quick patch, not a solution; soon enough it will simply stop doing anything.
If you are all so against bombs why did you create a new one?
Ask instead why using it as a bomb is the first thing that came to mind for a large number of people, some of whom felt compelled to make sure that all potential bombers would know about it as soon as possible.
Wouldn't it be easier to just put up a screen saying this page is no longer available?
Actually, no; this was quicker. We'd have preferred to do something else, but it was implemented in a way that made the seemingly obvious solution time-consuming. Rather annoying, actually.
-j
======================================================
SEPTEMBER 2 To ECWFRK
From:J Greely
Subject:Re: TO WEBTV Pt.2
To:ECWFRK@webtv.net (F T W)
Date:Wed, Sep 2, 1998, 7:55pm (EDT-3)
I'd be extremely annoyed if I was a legitimate user and my employer had to make me power off to tell me something is no longer available.
People who use it are in a hurry; they might not read a changed page, and might not have time to wait ten minutes to see if it worked.
Hopefully _very_ soon. I'd hate to see the newsgroups turn into a virtual mine field again.
That depends on what other silly crap we have to fix quickly. The right fix takes time. Shutting off all upgrades was the fastest way to stop people from grabbing internal builds, but it wasn't the best way; that took hours, and it was days before we were reasonably sure that we'd caught most of the people who got them. Shutting down Daily completely was the fastest way to keep people out, but it took several days to bring it back up in a way that we were reasonably sure was safe. We didn't need to ask.
Most people already know that it's an unfortunate fact that a lot of people get some sick thrill out of making life as difficult as possible for people.
...and others feel compelled to share secrets
regardless of the potential cost, so that they can stroke their own egos. Not an accusation to you personally, but definitely the case for some of the others.
Certainly there could have been a solution that would have been nearly as easy to implement as the way it was done that would not have given people the tools they needed to cause a disruption.
No. There were two choices: change the timeout, or build a completely new version of Tricks and make time in the QA group's schedule to test it. The latter will happen soon enough, but meanwhile we're not bleeding money into our phone bill.
What people do use those sort of tools ?
Stayin' Alive was created at the specific request of marketing employees who give demos of our products. It lets them set up their demo, cache critical pages, and hold the connection so that they don't have to leave a large group of people staring at a "connecting..." screen.
If volunteers, is it possible to sign up to become one like the previewers?
Previews has a form for people who want to sign up, and it includes a non-disclosure agreement.
Also, is there a liason where people like us could ask questions of that sort and report things that we found?
Not formally.
Some people have reported security issues through customer care, but due to training problems, some of those messages didn't get through to the developers and admins. We have someone working on that half of the problem now. I know the company actively discourages people doing this but many do it anyways because they are concerned about their own security. As many or more do it because they like doing what they're not supposed to do.
Witness the large number of messages posted recently that are more concerned about us taking away their holes than about how secure or reliable the service is. People who went to Daily to get Doom don't bother me a tenth as much as people who went there trying to find out new ways to violate our security model, but the folks who went in search of Doom shared their knowledge with the others, making them at least as guilty. Our honest users are very fortunate that the malicious ones didn't discover anything *seriously*
dangerous in their trips to Daily. The forced upgrade loop might seem a bit severe to get rid of internal builds, but the alternative was much worse. Stayin' Alive just costs us money; the internal builds could be used to cause real damage, and if something had happened, lawyers would be knocking on the doors of everyone we found who had gone to Willies. As it is, we would be quite justified in canceling
their accounts for violating the ToS.
Such as when, less than a year ago someone found out it was possible to access other peoples WebTV accounts, they reported it to WebTV, and then told everyone else that it could not be done until the breach was fixed.
Some of the people who were involved in that are remembered unfavorably; I was busy doing something else at the time, but in the past week a number of people have asked me if certain users were involved in the Daily problems.
Perhaps there should be a team of actual WebTV users, like the previewers, who would be authorized by WebTV to help seek out things like that.
It's a hard sell, especially right now. One of the reasons I've continued to read and respond to the newsgroups is that I'd like to know who falls into what category. No one has found a way that they could be used to disrupt anything so they have not been reported yet. There is concern over the search form trick, and it will be shut down. Getting to internal URLs directly *shouldn't* create a problem, but it has in the past, and some people have written code with the assumption that anything that can call an internal URL is "safe", and it takes a major code review to find them all. For that matter, it takes a lot of work just to find all of the input fields that can be used in this way. We know of two for sure, as well as a few others that would be vulnerable if it weren't for something else interfering, but the only way for us to devote staff time to finding all the possibilities is to pull people off of current development, which may delay the next release. All in all, it hasn't been fun recently.
-j
======================================================
SEPTEMBER 3 To ECWFRK
From:J Greely
Subject:Re: TO WEBTV Pt.2
To:ECWFRK@webtv.net (F T W)
Date:Thu, Sep 3, 1998, 2:58am (EDT-3)
I used my own judgement and didn't go any farther than the index screen.
A decision I wish more people had made. The last week and a half would have been more pleasant.
The other major problem is that the password wasn't discovered by one of the responsible people.
Perhaps, but they might not have learned any of the URLs hiding behind it without knowledge shared too freely. Computer security is a touchy business, and even among professionals you'll find strong disagreement on how and when to make things public.
But it is hard to do that because we don't know all the intracasies of the network and sometimes cannot predict what what trouble it could cause.
This is definitely clear with the Daily "explorers"; so far, no one seems to have figured out any of the reasons why we took it so seriously (hint: Doom ain't even on the list), and we're glad. I'd have shut down the entire service to stop that, if necessary, and kept it down until the situation was resolved. If anyone is caught with an internal build again, they'll be treated as a criminal.
For instance at least one person got into tricks about 2 months ago.
Yes, I know; at the time, it seemed to be an isolated case of some fool leaking the password, so we just changed it. I doubt that anyone guessed it this time, either; there are too many people who know it. This is one of the reasons why we're not concerned about the long-term implications of how it works today; the password-based security model for tricks is going to be replaced, and using Stayin' Alive as a bomb simply wouldn't work.
That was what we thought. I know of people who tried calling and telling people about something the've found but the person on the phone had no idea what they were talking about and so they basicly got a "I'll make a note of it" response and that was it.
We found trouble tickets in their database where someone had carefully entered complete details called in by a customer, and then closed the ticket without even trying to pass it to another group.
There is, however a third group of people, those who just wanted to see what was in there just out of curiosity of what else could be done and what will be available in the future.
The "in the future" part causes us problems, too, especially with things that involve licensing that might not be complete. Besides Doom, there are a number of changes coming, including new partners; spreading that around might upset the partners who are being replaced.
I'm not exactly sure what the internal builds are or what could have been done with them.
You're better off that way. We're changing it, so that leaking an internal build won't create as much risk as it does now (and even that is less than it used to be), but some things must remain if it's to be useful to us at all. I can't do my job without an internal build.
But as for the upgrade loop, most people who went there knew immediatly that they might screw up their boxes. In more than a dozen cases that I know of, the people who called to complain about their box being broken were not the people who had downloaded the internal builds. Mostly it was parents who didn't know why their username was gone or they couldn't get mail. Before things had been completely sorted out and the correct information passed to all groups, these customers had been told to mail in their boxes for fixing.
Also a suggestion. Couldn't you use the same security measures (or a variation of them) that are used to prevent unauthorised people from accessing the previewers pages and newsgroups be used for pages like tricks?
Well, all internal URLs were supposed to be protected from direct access, *except* for tricks (the person who accidentally broke that feature for 1.3 and 2.1 hasn't been found yet); the whole point of tricks was to put things that were useful to employees and licensees somewhere that they could be safely accessed directly. Things like sidestepping tricks to get to Willies weren't supposed to work.
The problem is that the search form trick is not the only way people are direct accessing intrnal URL's.
If there's another way that works with 2.2 and 1.4, we don't know about it yet, or at least not everyone involved does (I'm not even in the same city as our developers, which makes it complicated to stick my head out of the office door and yell at them). Both of those builds should not allow direct access to anything except wtv-tricks unless the link is embedded in a trusted page.
That's the best reason to have some of the users to look for things like that. To free up resources that would be better used in another area.
Users can exercise many of the code paths and templates, but not all; as a simple example, you can't tell me if the french version of the Explore tree has a hole in it, or if the tiles on next Saturday's home page are safe. You definitely can't tell me if the slightly different home page for a box running 2.2.5 is safe (or even 2.1.5, for that matter; you simply don't have the right hardware).
-j
======================================================
SEPTEMBER 3 To ECWFRK
From:J Greely
Subject:Re: TO WEBTV Pt.2
To:ECWFRK@webtv.net (F T W)
Date:Thu, Sep 3, 1998, 4:50am (EDT-3)
One thing to add. The password wasn't leaked from inside the company this time. The person who distributed the password told some people how it was found. The person who found it simply found seqret on a list of commonly used >passwords and tried a few varitations.
Actually, we know that it was spread by at least one former employee, although that one *was* sufficiently stupid that it was guessable. My group wasn't in the loop for that change, and when we changed it to something reasonable, people complained. This is the primary reason that we're changing the model completely; people simply don't like good passwords.
Although I do hope that some games will be added soon. I for one wasn't too thrilled with the prospect of Doom (just about everyone has already played it to death anyway) but I was kind of excited about "You Don't Know Jack!".
Doom is a convenient proof-of-concept game; the source is readily available, it's already been ported to multiple platforms, and it has good name-recognition value. YDKJ, which didn't seem to be nearly as exciting to the "explorers", is the sort of thing that would probably work better in practice, since we already do daily downloads for things like TV listings, so we could update questions frequently.
Also what is it that you do at WebTV? Just curious.
I'm one of the folks who runs the production service. Among other things, I'm the one who handles the weekly downtimes, installs service-side changes, and configures new machines to expand our capacity. I also write tools and documentation for the above.
-j
======================================================
SEPTEMBER 3 To ulTRAX
From: jgreely@corp.webtv.net (J Greely)
Date: Thu, Sep 3, 1998, 2:03pm (EDT-3)
To: ulTRAX@webtv.net
Subject: Re: your mail
In looking at the vast number of commands that WTV uses, I have always wondered why they were all needed.
Why the service they provide is needed, or why it's separate from something else?
(Though I have to wonder why it's necessary to have 2 ways to get to the Home Page wtv-home:/home and >wtv-tricks:/home)
That one is actually one of the original reasons for having tricks in the first place. Except when the client is broken (sigh), an external web site can't send you to the home page; it was considered desirable for our partners (such as Excite) to be able to have a home link, and a few other things.
But there are Client, Cache, File, Tricks, ROM, ROMCache, Disk, etc commands. What purposes do these grouping serve?
Each one identifies a different place to get data from. It sounds like you already understood this, so I'm not sure what you're asking.
Since I was the one who went to WTV...
You weren't the one they were asking about; there were apparently other names that became known in connection with it.
-j
======================================================
SEPTEMBER 9 In Response to post by ulTRAX
From: jgreely@corp.webtv.net (J Greely)
Date: Wed, Sep 9, 1998, 5:48pm (EDT-3)
To: ulTRAX@webtv.net (Killer<> ulTRÅX <> Willie) Subject: Re: WTV-TRICKS:/INFO EZ TESTING
ulTRAX@webtv.net writes: Tech Info variable #21 debug/nondebug Hey jgreely... if you're reading
this... can you give us any clues?
People who got to Killer Willies should be able to figure that one out.
-j
======================================================
SEPTEMBER 9 To ulTRAX
From: jgreely@corp.webtv.net (J Greely)
Date: Wed, Sep 9, 1998, 6:34pm (EDT-3)
To: ulTRAX@webtv.net
Subject: Re: WTV-TRICKS:/INFO EZ TESTING
Jgreely, you're no help.
Actually, I thought that response would be very helpful to anyone who took notes when they hit Killer Willies. That one was pretty obvious.
Actually there may be as many as 30, not 21. That's going to be a lot of guesswork. You know how much trouble you could save me by just sending over some spare WTV Tech Manuals?
I don't have any. In fact, I've never even *read* tech manuals for the client software, much less seen the source. I know a great deal about the server side, but even there there are large chunks of the internals that I know nothing about, because they haven't broken recently. What I do is primarily "black box" debugging; if the server or client doesn't give me an error message, and the failure is not obviously revealed by watching the output of a debugger, I pass it off to the team that wrote it. Then I go off and try to work around it until they come up with a fix.
I take it the connection speed is measured at login and is no represention of what the rate is at any other time.
It should be the speed that your connection was at when you authenticated.
Some Questions about 2.2. When I was a previewer I absolutely hated 2.2. My mail service went to hell.... difficulity in trying to post, write, or forward. (and I have no links in my sig) Couldn't get into Mail Storage for nearly a week..... all made worse by constant freeze ups.... when the Publisher has kept me waiting for more than a minute. Some things haven't changed.
Hmmm, I was running 2.2 for quite a while before it got to Previews, and never saw any of this. The only problem I had regularly (apart from the known bugs) was that it would freeze up completely about once every 10 days, and I'd have to unplug it.
But, what's happening when I'm trying to write or post? The green load bar shows a complete load... yet the letter form doesn't show up.
Your problem may not be on the client; does this tend to happen at roughly the same time each day?
I'm fond on saying that being a previewer was one of the most miserable experiences of my life.
It could be worse; you could use Daily, where the whole service breaks in a different way every day.
-j