MoparScape 508 Update Server

#1

Here are all of the sources I am releasing for MoparScape 4, so far:
http://cache.hybridscape.com/MoparScapeSources.jar

As you can see, there is a 508 Update server that serves up the cache, which works well, except about half way through it starts disconnecting the client every 4 or 5 requests, causing it to keep reconnecting. You can see this behavior by running the client in debug mode:

http://www.moparisthebest.com/moparscape4.php?version=508&debug=true

Can anyone suggest a fix for (preferably) the server or even in the client? I’ve tried about everything so far and can’t come up with one. I’d appreciate any help at all.

#2

did you try reading fully from the http location then sending to the client?

#3

It would time out and send me to that error page if I tried that, and it shouldn’t change anything. The same thing happened when hosting the cache running a server on localhost, so it isn’t a connection speed issue.

#4

Great, I just discovered a new problem. When I log into a 508 server, I get this:

UpdateServer: Error, http://cache.hybridscape.com/508/7/-32165 does not exist. UpdateServer: No alternatives, continuing. UpdateServer: Error, http://cache.hybridscape.com/508/7/-23781 does not exist. UpdateServer: No alternatives, continuing. UpdateServer: Error, http://cache.hybridscape.com/508/7/-31671 does not exist. UpdateServer: No alternatives, continuing. UpdateServer: Error, http://cache.hybridscape.com/508/7/-31562 does not exist. UpdateServer: No alternatives, continuing. UpdateServer: Error, http://cache.hybridscape.com/508/7/-31561 does not exist. UpdateServer: No alternatives, continuing. UpdateServer: Error, http://cache.hybridscape.com/508/7/-31560 does not exist. UpdateServer: No alternatives, continuing. UpdateServer: Error, http://cache.hybridscape.com/508/7/-31559 does not exist. UpdateServer: No alternatives, continuing. UpdateServer: Error, http://cache.hybridscape.com/508/7/-31558 does not exist. UpdateServer: No alternatives, continuing. UpdateServer: Error, http://cache.hybridscape.com/508/7/-31544 does not exist. UpdateServer: No alternatives, continuing. UpdateServer: Error, http://cache.hybridscape.com/508/7/-27386 does not exist. UpdateServer: No alternatives, continuing. UpdateServer: Error, http://cache.hybridscape.com/508/7/-32183 does not exist. UpdateServer: No alternatives, continuing. UpdateServer: Error, http://cache.hybridscape.com/508/7/-32182 does not exist. UpdateServer: No alternatives, continuing. UpdateServer: Error, http://cache.hybridscape.com/508/7/-32181 does not exist. UpdateServer: No alternatives, continuing. UpdateServer: Error, http://cache.hybridscape.com/508/7/-32180 does not exist. UpdateServer: No alternatives, continuing. UpdateServer: Error, http://cache.hybridscape.com/508/7/-32179 does not exist. UpdateServer: No alternatives, continuing. UpdateServer: Error, http://cache.hybridscape.com/508/7/-32166 does not exist. UpdateServer: No alternatives, continuing.

But of course cache index 7, id -32165 doesn’t exist. In fact I don’t think any negative IDs exist, anyone know the problem here?

I guess I must be dumping the cache wrong, anyone mind looking at the cache dumper and seeing what the problem is?

#5

it looks like the client is sending a byte for the priority/‘type’ (one or zero, which you read correctly), and then three bytes, which you read as a short. perhaps that’s causing the problem.

#6

Do you know how those 3 bytes are supposed to be read/interpreted?

Also, does anyone know if there is a way to find out for which index/id that data exists for without bruteforcing it or logging what the client requests? I apparently haven’t dumped the ENTIRE cache because I thought that the id could only go up to MAX_SHORT, when apparently it can go up to MAX_INT. (which is taking hours upon hours)

#7

this is the method used to send them in the client (refactored a bit):

public void method918(int i) { buffer[currentOffset++] = (byte) (i >> 16); buffer[currentOffset++] = (byte) (i >> 8); buffer[currentOffset++] = (byte) i; }

and to read them you could probably do something like this:

int id = (in.readUnsignedByte() << 16) + (in.readUnsignedByte() << 8) + in.readUnsignedByte();

oh, and you could probably find out where the files go up to by looking at the reference table data for each cache (the files in file system 255 besides file 255).

#8

[quote=“Metho D, post:7, topic:331146”]this is the method used to send them in the client (refactored a bit):

public void method918(int i) { buffer[currentOffset++] = (byte) (i >> 16); buffer[currentOffset++] = (byte) (i >> 8); buffer[currentOffset++] = (byte) i; }

and to read them you could probably do something like this to read it:

int id = (in.readUnsignedByte() << 16) + (in.readUnsignedByte() << 8) + in.readUnsignedByte();

I changed my:

int id = in.readShort();

to your:

int id = (in.readUnsignedByte() << 16) + (in.readUnsignedByte() << 8) + in.readUnsignedByte();

Which apparently isn’t right because it never serves up anything before throwing a:

Server error: java.io.EOFException at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:273) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:78) at org.moparscape.userver.Server.run(Server.java:64)

Honestly I don’t know much to anything about the format, I’m using code I was supplied with by Silabsoft I believe, any hints on where to find that info and/or how to look it up?

#9

oh, whoops. the data you’re supposed to read is a combination of the cache index and file number put into three bytes. try this instead:

int uid = (in.readUnsignedByte() << 16) + (in.readUnsignedByte() << 8) + in.readUnsignedByte(); int index = uid >> 16; int id = uid & 0xFFFF;

as for the reference tables, anInt454 in Class21renamed.method349 might be what you’re looking for (it should be the entryCount field, but i don’t remember if it represents the complete count of files in each cache).

#10

I’ve changed the code as follows:

//int index = in.read() & 0xff; //int id = in.readShort(); int uid = (in.readUnsignedByte() << 16) + (in.readUnsignedByte() << 8) + in.readUnsignedByte(); int index = uid >> 16; int id = uid & 0xFFFF;

And, while it actually works, it seems to work the same as the original code. It works, but it disconnects a lot, so something is still not right. I’ve been pulling my hair out for months over this.

A count of files in each cache won’t do much good, as they aren’t all in a row. Sometimes it skips quite a few numbers, here is an excerpt from the dumplog.txt:

data for: 6,0 data for: 6,2 data for: 6,76 data for: 6,85 data for: 6,122 data for: 6,157 data for: 6,177 data for: 6,327 data for: 6,552 data for: 6,715827884 data for: 6,715827958 data for: 6,715827967 data for: 6,715828004 data for: 6,715828039 data for: 6,715828059 data for: 6,715828209 data for: 6,715828434

#11

do any errors pop up in the client when this happens or does it just disconnect? the only thing that comes to mind right away would be sending some invalid header data for the files or something like that.

#12

No errors on the client, other than a noticeable slowdown, but if you run in debug mode you can see these errors, but I printed out the requests here (508/index/id format)

508/255/255 508/0/2 508/0/3 508/0/4 508/0/5 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/20/3 508/21/3 508/22/3 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/21/3 508/22/3 508/2/4 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/22/3 508/2/4 508/16/4 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/2/4 508/16/4 508/17/4 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/16/4 508/17/4 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/17/4 508/18/4 508/19/4 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/18/4 508/19/4 508/20/4 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/19/4 508/20/4 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/20/4 508/21/4 508/22/4 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/21/4 508/22/4 508/2/5 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/22/4 508/2/5 508/16/5 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/2/5 508/16/5 508/17/5 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/16/5 508/17/5 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/17/5 508/18/5 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/18/5 508/19/5 508/20/5 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/19/5 508/20/5 508/21/5 Server error: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.moparscape.userver.v508.OndemandServer.handleConnection(OndemandServer.java:171) at org.moparscape.userver.Server.run(Server.java:64) 508/20/5

So I don’t believe the header information is wrong, since I dump it the same for every file, and the fact that the files end up getting served and accepted by the client, it justs requests them multiple times before it gets accepted.

I know 508 tries to connect to port 443 as well, so I tried running a server there as well, but it seemed to use a different protocol of some type and always disconnected. I don’t know if this has anything to do with anything, but I’m out of ideas.

#13

How many requests are you serving at one time? you should send no more than 10 requests at one time.

#14 
    public void run() {
        thisThread = this;
        while (isRunning)
            try {
                handleConnection(sSock.accept());
            } catch (IOException e) {
                handleException(e);
            }
    }

I just handle them one at a time, as they come in. Do they send multiple requests at once?

#15

sends up to 10 file requests and then listens to receive.

#16

That’s going to take a decent amount of code change, I’ll start working on it. While we are on the topic though, does 317 do the same thing?

#17

317 can receive up to 100 files at one time. and then sends more requests.

#18

So I hacked up some code to handle 10 requests at a time, changed the previous run method to this:

    private synchronized int modThreadCount(int mod) {
        threadCount += mod;
        return threadCount;
    }

    private int threadCount = 0;

    public void run() {
        thisThread = this;
        while (isRunning)
            try {
                //handleConnection(sSock.accept());
                if (modThreadCount(0) < 10) {
                    (new Thread() {
                        public void run() {
                            System.out.println("threadCount: " + modThreadCount(1));
                            try {
                                handleConnection(sSock.accept());
                            } catch (IOException e) {
                                handleException(e);
                            }
                            modThreadCount(-1);
                        }
                    }).start();
                } else {
                    Thread.sleep(500);
                }


            } catch (Exception e) {
                handleException(e);
            }
    }

I’ve changed the value of Thread.sleep() to 1000, 500, and 50. But all seem to have the same results, it still disconnects. Any ideas? (or anything obvious I did wrong?)

#19

what the hell not 10 connections, 10 file requests. get on irc…

#20

Well I’ll be a son of a bitch, it actually does make multiple requests at once (more than 10 though, wonder how many…)

                    int uid = (in.readUnsignedByte() << 16) + (in.readUnsignedByte() << 8) + in.readUnsignedByte();
                    int index = uid >> 16;
                    int id = uid & 0xFFFF;
                    System.out.println("request0: " + String.format(odsPath, index, id));
                    // if we aren't requesting update keys
                    if (index != 255) {
                        for (int x = 1; x < 15; ++x) {
                            uid = (in.readUnsignedByte() << 16) + (in.readUnsignedByte() << 8) + in.readUnsignedByte();
                            index = uid >> 16;
                            id = uid & 0xFFFF;
                            System.out.println("request" + x + ": " + String.format(odsPath, index, id));
                        }
                    }

Result:

request0: 508/0/17
request1: 508/0/0
request2: 508/18/0
request3: 508/0/4864
request4: 508/0/21
request5: 508/0/0
request6: 508/22/0
request7: 508/0/5888
request8: 508/0/24
request9: 508/0/0
request10: 508/25/0
request11: 508/0/6656
request12: 508/0/27
request13: 508/0/0
request14: 508/28/0
UpdateServer: Error, http://cache.hybridscape.com/508/28/0 does not exist.
UpdateServer: No alternatives, continuing.
request0: 508/29/0
request1: 508/0/7680
request2: 508/0/31
request3: 508/0/0
request4: 508/34/0
request5: 508/0/9472
request6: 508/0/40
request7: 508/0/0
request8: 508/43/0
request9: 508/0/11264
request10: 508/0/46
request11: 508/3/0
request12: 508/0/259
request13: 508/2/29185
request14: 508/3/627

Now I just need to rewrite this to queue the requests then serve them up. Damn it…