Loading huge maps

[TheLostGuru]

I am basically trying to create a simple scrolling tilebased map, but the map is going to be HUGE! It will only be involve horizontal scrolling. I am wondering what is the best way to store and load in a large amount of variables. Should I go with XML? I would also like to be able to devise a system where people couldn't steal my map data, either from wherever I host the XML file or from the game itself. Is this possible? And if so, what steps do I need to take?

[mr_malee]

I like xml, but if you don't want people to see it you can always go with a .as file but that requires re-compiling when mapData changes.

also with a .as file you can write the map as an array and there's no need to parse it, unlike xml, which you would have to do.

as for loading them:

something like this perhaps:

var x, y, maxX, maxY, mapX, mapY;

function startLoad(){

	x = 0;
	y = 0;
	maxX = 0;
	maxY = 0;

	//load and parse xml or .as

	mapX = map[0].length;
	mapY = map.length;
}

function loadMap(){

	maxX += 10;
	maxY += 10;

	maxX = Math.min(mapX, maxX);
	maxY = Math.min(mapY, maxY);

	for(y=y; y<maxY; y++){
		for(x=x; x<maxX; x++){

			//create tile
		}
	}

	if(maxX == mapX && maxY == mapY){

		//start game
		switchEnterFrame(game);
	}
}

function switchEnterFrame(func){
	
	onEnterFrame = func;
}

function game(){
}

startLoad();
switchEnterFrame(loadMap);

[renderhjs]

it´s not like looking at the xml file will make anyone smarter -i´d go for xml since it´s easier to develop with and debug.

[Dragonusthei]

why could you not just get php to move the xml to a random location and feed that location into flash. That way the only full time place the XML would be stored would be on the non public files directory. That way the XML would be as secure as other sites mysql information.

[rachil0]

If you have reaaalllly huge maps, maybe you could stream them in from disk. Think of it as a 1D linked list, where each node is an XML file that represents a largeish chunk of map (maybe 5-10 screenlengths worth). Each XML node has a special field which represents the "next" node/ XML file to stream, connecting them in a fixed sequence. (Maybe they keep links to the "previous" node too, if you want the ability to walk both left and right).

While playing, only keep a single node in memory. When the player encroaches upon the end of the node, at which point you peek inside the XML and load up the next node and write its contents into your tilebuffer, etc.

It's just like a linked list, except instead of it being a lot of nodes in memory that are connected by pointers, it's a lot of nodes on disk that are connected by filenames. This should be able to scale up to huge maps - your world could take as much space as you can store (ie your map could be larger than what you can fit in system ram). I've never done something like this because I've never had such a staggering amount of content, but I think it should work.

[Sauceofallevils]

Use a similar method to Strille's method, it seems very effective for large maps, just make it only capable of scrolling horizontal.

Tutorial File and Fla Download:
http://www.strille.net/tutorials/part1_scrolling.php

Engine Demo:
http://www.strille.net/works/misc/platform_demo/

[Flashkit]

I too am working on a large map som 400 x 400 in size.
I started off using xml but it got way too large. I used tiled to make the maps, then I convert the map into a single LONG array of numbers, that keeps the size down and makes access quite fast.

I have heard of a way of storing the tilemap as a very tiny image, and then each pixel corresponds to a tiel object you just query the image in the range that you are at. I havent tested this approach

[rachil0]

In that case you'd still have to break it down into multiple images, I think bitmapDatas have a maximum width and maximum height of 2880 pixels. That's still a bunch of tiles though, and would be very space efficient I think.

[renderhjs]

@Flashkit:
not sure how you did it but many people tend to bloat their xml with tags instead of attributes or string chains.
Also if you have a set of attributes for each tile a Index system would help alot not just simplifying the file but also reduce lots of space. example:

<map prefixTex="tiles/"> 
    <id> 
        <tile id="w"     cost="0,2"     tex="water_1.gif,base_1.gif"    z="0"/> 
        <tile id="b"     cost="2"     tex="base_1.gif"        z="0"/> 
        <tile id="g"     cost="2"     tex="gras_1.gif,tank_renderings.jpg"    z="0"/> 
    </id> 
    <grid> 
        <r>b,b,b,w,w,w,w,g,w,b</r> 
        <r>g,w,w</r> 
        <r>g,b,w</r> 
        <r>g,b,w</r> 
        <r>g,b,w</r> 
        <r>w,w,w</r> 
        <r>g,b,w</r> 
        <r>g,w,w</r> 
    </grid> 
</map> 

example from this thread
what I like on xml (the way I do it) is that I can easy and quickly edit common things that need to be played with in the beginning. In the end of course I do have as well a propper editor that allows bigger and more complex maps but in the meantime it is very helpfull for me to have a xml file that serves me with the variables I need for the map without compiling each time the flash file,- almost like many triggers

[Squize]

For a huge map like 400x400 xml is too bloaty.

Counting 2 bytes per tile ( So tiles 0-99 stored in ascii ), that's 400x400x2 = 320,000 bytes in a raw file. That's without giving any tile an attribute ( Biggest map I've used was 200x200, and that took 195k ).
Plus Flash would struggle like hell to parse that much data ( Pre F9 anyway ).

A happy medium is to have a "master" xml file, with info about the levels, and a link in them to the raw data, so that xml file has the human readable stuff in there, and the map data is left as what it is.

The way of storing map data as a png is great, 'cause then you get the advantages of the png compression ( It's like storing important vars in an image which sits on your server as a way of protecting your game ).

Squize.

[Alluvian]

I love this idea of of a graphic storing tile data as pixels. I assume there would be a separate xml that holds the legend that knows what color equates to what attributes?

Would png compression cause issues by changing the color slightly? Or would you just convert the hex color string to a number and take the closest match for each tile?

Or is the idea to store it as a lossless image? Would that have much space benefit? I guess it depends on the map layout. GIF would compress large areas of ocean or flat plains nicely. You could do that in a textfile/xml by setting up a way to 'paint' large regions, but the GIF compression would do all that work for you.

[Squize]

png is lossless, and compresses better than gif ( In most cases ).

You wouldn't really need xml as a legend / look-up, 'cause things like that are hardcoded in your game ( You wouldn't need to change say the value of a water tile from 10 to 100 in an xml file unless you're making a really generic engine to give to say a designer ).

If you limit your number of tiles to 255 ( Which is loads ), then the red channel can be the tile value, the green attributes for it and blue is a spare for when you think of a good use for it

Squize.

[rje]

I'm building a game with 600x600 tiles, 4 layers. I developed a map editor tool chain based on the open source Tiled map editor - http://mapeditor.org/

Tiled outputs XML - lots of it (to be precise over 60Mb for 600x600x4 layers). I developed a Java application that processed this XML into a .as that contained a single array with 1 encode int that represented a single tile - all 4 layer were encoded using bitwise operations. I just compile it into the project. The file is ugly, full of things like:

_data.push( 0x10002,0x20001,0x20102,0x20102...

But its all managed fine in the map editor. Btw I don't know how efficient the AS3 xml parser is, but the default java one allocated >600Mb heap space to process this file, I dont think you want that going on in the flash player.

To summarize - design it using whatever you feel comfortable in XML etc. Just transform it into something usable for the flash player. You could use any number of the scripting languages out there to do this - even XSLT.

Good luck.

[renderhjs]

@rje: how big is the compiled swf compared to the xml?, and what about loading such a big array- any lags?

[rje]

I have a demo level I'm working with, 2 layers 120x100. The XML (in the Tiled format, not the most compact) is about 500Kb. The transformed .as file is about 50Kb. The entire compiled .swf is <30Kb. About 28Kb of this is the engine.

I just did some quick sums - it seems the array is very tightly packed on average 1 byte per tile (this level has a lot of empty areas, maybe thats why).

I don't notice any delay on loading. Maybe if you have millions of tiles, but nothing with 1000's.

[renderhjs]

those are nice results,- but still the workflow not yet optimized. I´m planning on working on a complex engine sometime soon so this thread already gave me some ideas.
I am thinking of cascading elements - like instanciating certain constructions and just call that instance a few times in the map file.

I am also thinking of brush tiles where I only store the draw path and calculate the tiles from the brush tip along the path on the fly.

Another idea is to stream level areas as they are needed- pretty much as like what they did with God of War for the playstation. Similar to a tileengine that scrolls tiles- the same could be done with map areas.

Also if the String and Array class are fast enough in AS3 XML might be still a choise with strong compression or otherwise some ByteData that requires less character ID space

[rje]

brush tiles - nice idea, might be quite a bit of overhead during run time, also you'll need good tooling to help design levels (maybe a SVG package like inkscape?)

If your worried about size/memory then you could use the ByteArray class (as3). It supports zlib compression, although you have to compress/decompress the the entire array + I think its slower than Array.