Currently the developers are putting their own money into JC2-MP to keep the servers online.

Please take a few seconds of your time and disable your AdBlock plugin for our website.

Ad revenue is not going to developers, it is used purely for covering our hosting costs.

 

You are also free to donate, which removes all ads from our website!

Patch 0.3 was just released! Full changelog here: https://t.co/4A50m6IKen

2 years ago

Advertisement
September 21, 2019, 07:59:51 pm

Author Topic: Basic NPC AI  (Read 15718 times)

SinisterRectus

  • JC2-MP Betatester
  • Sr. Member
  • *****
  • Posts: 451
    • View Profile
Re: Basic NPC AI
« Reply #30 on: June 11, 2015, 06:13:32 am »
I was able to fix the height issue that I described above for the MHC and went ahead and collected nodes for (almost) the entire game world.

I quickly learned that raycasts do not work on terrain that isn't loaded, and terrain only loads in cells that are near to the player. So, the mapping process became this: Teleport to a cell; wait for the terrain to load; build a map of the cell; send the data to the server; save the data to a file.

Teleport to a cell - This part was easy. Find the center of a cell and teleport to it.

Wait for terrain to load - This was a little tricky. I figured out that when a player teleports to an unloaded location, the player position updates, but the player does not load until after the terrain does. Conveniently, player velocity is zero while it is loading, so checking whether the terrain had loaded became a matter of waiting until the player had a non-zero velocity after teleporting.

Build a map of the cell - For this, I used my method for finding nodes, except with the cell dimensions as the boundaries.

Send the data to the server - I originally wanted to save cells of size 512, like the native cells, but I was prevented from sending such large tables to the server. I ended up going with 128 instead. This split the map into 65,536 cells, and out of all of these cells, only 2 were too large to send to the server. I'll need to make some adjustments to make them work.

Save the data to a file - I decided to write the data straight to .lua files. Doing this generated a whopping 84.0 GB (78.2 GiB) of text spanning 65,534 files. They range from 8.07 MB (7.69 MiB) to 1.09 MB (1.04 MiB), with an average of 1.28 MB (1.22 MiB). The files compress to about 5% size using 7z (LZMA) and 10% using zip (deflate).

The whole process took 35 hours, far longer than my original estimate of 5 hours due to the time it took to teleport Rico around Panau. And this was just for obtaining the node vertices. I have not yet added the edges/connections, which will make the file sizes even larger. I'm going to see what I can do to make the data more manageable. This will probably involve segregating sea data from land data.
« Last Edit: June 11, 2015, 06:54:58 am by SinisterRectus »

Darwood37

  • Donator
  • Hero Member
  • *****
  • Posts: 690
    • View Profile
Re: Basic NPC AI
« Reply #31 on: July 04, 2015, 04:36:50 pm »
Couple of ideas that may speed up the process for you or maybe you have already done this. I am very impressed with this and was amazed when i visited your test server.

Is it possible to have Rico automatically teleport to the center of each cell and once he has spawned in(non zero velocity) start the data gathering. If you can check if this data gathering has finished then have him spawn to the next cell.

If you set the stream distance to the size of the cells you are gathering data from maybe that would speed up things as well?

Also if you need to update the data for any reason can you have two servers running? One that hosts the game and the other that is local only and just collects the data and passes it to the server hosting the game.

just my two cents worth on a very exciting project.



SinisterRectus

  • JC2-MP Betatester
  • Sr. Member
  • *****
  • Posts: 451
    • View Profile
Re: Basic NPC AI
« Reply #32 on: July 04, 2015, 05:38:48 pm »
Thanks Darwood. I've actually put this project on hold for a bit. I've been working on some other things and I am also waiting until after the next release of JC2MP to continue working on the NPC logic.

Is it possible to have Rico automatically teleport to the center of each cell and once he has spawned in(non zero velocity) start the data gathering. If you can check if this data gathering has finished then have him spawn to the next cell.
Re-read my post above, that's exactly what I did. Unfortunately, I messed up and didn't store the height data with enough precision. The filesizes were also too large, so I tossed all the data and am working on re-gathering it with more precision and in a different format. Nodes first, then edges.

Quote
If you set the stream distance to the size of the cells you are gathering data from maybe that would speed up things as well?
As far as I know, JC2MP stream distance doesn't affect how terrain loads, only how API objects are streamed in and out.

Quote
Also if you need to update the data for any reason can you have two servers running? One that hosts the game and the other that is local only and just collects the data and passes it to the server hosting the game.
You can't update the data without a client. I suppose one could run a dedicated client, but I'm not sure how reliable or practical that is.
« Last Edit: July 04, 2015, 05:40:48 pm by SinisterRectus »

tally

  • Full Member
  • ***
  • Posts: 121
    • View Profile
Re: Basic NPC AI
« Reply #33 on: July 13, 2015, 08:36:57 pm »
I have managed to build one of the bikes, the only thing is that you have to move all of the parts relative to each other, which is a pain, also, natural movement might be hard to replicate.

SinisterRectus

  • JC2-MP Betatester
  • Sr. Member
  • *****
  • Posts: 451
    • View Profile
Re: Basic NPC AI
« Reply #34 on: July 14, 2015, 06:28:33 pm »
I have managed to build one of the bikes, the only thing is that you have to move all of the parts relative to each other, which is a pain, also, natural movement might be hard to replicate.
Yeah, vehicles are a different beast compared to players. With players I can record bone positions and angles. With vehicles, the positions and angles have to be manually constructed.

SinisterRectus

  • JC2-MP Betatester
  • Sr. Member
  • *****
  • Posts: 451
    • View Profile
Re: Basic NPC AI
« Reply #35 on: August 16, 2015, 04:36:25 am »
Just wanted to let you all know that I have not given up on this project, despite the lack of updates. I've actually been working on it regularly, testing out some of the great upcoming features that will be in the next release of JC2MP! I'll have plenty of stuff to share here after it comes out.

[email protected]

  • Jr. Member
  • **
  • Posts: 50
    • View Profile
Re: Basic NPC AI
« Reply #36 on: August 16, 2015, 05:05:01 pm »
encore!

Perky

  • Newbie
  • *
  • Posts: 10
    • View Profile
Re: Basic NPC AI
« Reply #37 on: September 02, 2015, 06:14:43 pm »
This thread got me thinking if it was possible to extract all the road co-ordinates from the game files, so at least you could have NPC cars move on 'rails'. However I can't seem to figure out the co-ordinates.

In world.xml (world.bin) you can see a bunch of world locations each with 2 co-ordinates. I assume these co-ordinates specify an AABB rectangle. There is also a reference to a file. So I looked inside 'roads/roo_etno02' and inside that is a .blo file that can be converted to .xml:

https://gist.github.com/anonymous/218dfee0af4dcb8a9d70

Inside that are a list of road segments, there is a list of co-ordinates that specify the road shape, however these co-ordinates have 4 numbers each, x, y, z and a 4th number between 0 and 1.

I've checked these out by drawing them as lines in the game and they definitely draw the shape of roads but in the wrong position.

I also wondered if you could extract the heightmap. There is a file called terrain_win32.dat, I think that specifies the heightmap of the terrain. There's an article on Gamasutra where JC devs explain they store the heightmap in 16bit value which are then converting to floating point values with 3bit exponent and 6bit mantissa:

http://gamasutra.com/view/feature/192007/sponsored_the_world_of_just_cause_.php?page=3

SinisterRectus

  • JC2-MP Betatester
  • Sr. Member
  • *****
  • Posts: 451
    • View Profile
Re: Basic NPC AI
« Reply #38 on: September 03, 2015, 01:54:55 am »
Great find, Perky! I don't know why I didn't think to check the game files first.

The first set of coordinates are most definitely a bounding-box, and the road files most definitely contain road coordinates (note that they are relative to the <value name="world" type="mat"> position).

Here is the Three Kings area. It seems that bridges and a few roads are missing, and I'm not sure what's going on around the hotel itself, but it's mostly there.





----

And I've seen that article before. It's kind of crazy how they handle terrain. I'm not sure whether it can be extracted, but even if it could be, there'd still be the issue of buildings and other objects to handle. I have actually finished generating a 2 meter resolution map of Panau, buildings included. I will talk more about it at a later date.
« Last Edit: September 03, 2015, 03:46:09 am by SinisterRectus »

Trix

  • Developer
  • Full Member
  • *****
  • Posts: 216
    • View Profile
Re: Basic NPC AI
« Reply #39 on: September 03, 2015, 08:39:11 am »
Great find, Perky! I don't know why I didn't think to check the game files first.

The first set of coordinates are most definitely a bounding-box, and the road files most definitely contain road coordinates (note that they are relative to the <value name="world" type="mat"> position).

Here is the Three Kings area. It seems that bridges and a few roads are missing, and I'm not sure what's going on around the hotel itself, but it's mostly there.





----

And I've seen that article before. It's kind of crazy how they handle terrain. I'm not sure whether it can be extracted, but even if it could be, there'd still be the issue of buildings and other objects to handle. I have actually finished generating a 2 meter resolution map of Panau, buildings included. I will talk more about it at a later date.

The game objects and other assorted placeables come from the CGeometryObject definitions in the location's BLO file from within its NLZ archive.

SinisterRectus

  • JC2-MP Betatester
  • Sr. Member
  • *****
  • Posts: 451
    • View Profile
Re: Basic NPC AI
« Reply #40 on: September 22, 2015, 04:27:43 am »
Time for a massive progress update!

First and foremost, I finished building a map of Panau. My previous attempt at this generated 84 GB of data, which was only a fraction of what the final product would have been. Since that was already too much, I tossed it all and spent some more time squeezing down the file sizes. I lost count somewhere along the way, but I know that I spent hundreds of hours and billions of raycasts scanning and re-scanning Panau before I settled on a method that worked.

I still broke down the map into cells of 128x128 meter size; however, I switched from 1 m resolution to 2 m resolution in order to reduce file sizes and resource usage. With some advice from SK8RJOSH and Trix, I also realized that saving data for sea-level nodes was not necessary, so I spent the time figuring out which cells contained nothing above sea-level. It turns out that out of 65,536 cells, 30,165 (46%) contain no objects or terrain above sea-level, 26,190 (40%) contain only objects or terrain above sea-level, and 9,181 (14%) contain a mix of both. Thus I was able to save data for only 35,371 cells (54%) and ignore the rest. For cells that do not have data saved, pathfinding grids can be procedurally generated at sea-level.

Let's look at an example cell: 18x16, which is located at Lost Island.
- To map and store its 4390 nodes requires 12,235 raycasts over 240 ms, and 1200 KB memory.
- Processing its 4-direction edges requires 17,024 raycasts over 400 ms, and 200 KB memory.
- In total, that's 29,259 raycasts over 640 ms, and 1400 KB memory.
- The node file is 64 kB; the edges file is 124 kB. I have not yet combined them into one, but it should be about 188 kB combined.
- Loading the same cell from files to the server takes about 330 ms. The cell is loaded from files faster than it is built with raycasts, which is nice.

My most recent scan of Panau took 29 hours and 20 minutes (695 minutes for the nodes, 1065 minutes for the edges), down significantly from what was originally 70-80 hours. Altogether, the data is 6.08 GB (5.66 GiB). That's an average of 172 kB per cell considering the 35,371 cells that actually have files, or 93 kB per cell if you consider all 65,536.

For your viewing pleasure, here is a visualization of the cells loading and unloading from files. Notice how the procedurally generated cells over open water load more quickly.

If you've been reading carefully, you'll notice that cell 18x16 is about an average sized cell, and uses 1.4 MB memory. Multiply this by 65,536 and you get 92 GB, a lot more than your average server can handle. Obviously loading every cell onto the server is out of the question, so when I was working on my AI module, I wrote a script that loads cells when an NPC spawns, and unloads the cells if they are not in use, which depends upon NPC and player density. Obviously, with enough active NPCs and players, you could still run out of memory, but with a bit of diligence, you can prevent this from realistically happening.

Lastly, I just wanted to show off my climate map. By using LocalPlayer:GetClimateZone() at the center of each of my 128x128 cells, I was able to serialize a climate map of Panau as a Lua table, which is 193 kB in file format, and can be utilized on a server. Here is the data visualized. White is arctic, yellow is desert, bright green is jungle, dark green is grass, red is city, and magenta is default (whatever that is). There are a few places where the climates are weird, maybe due to the low resolution, but it is mostly accurate. I use it to spawn civilian NPCs with models based on climates.



----

As far as the NPC AI goes, I guess the best way to explain my progress is with the few videos I've taken over the past three months. I'll post them below with short descriptions, and if you have comments or questions, feel free to post them:

Video 1 - The first thing I did when ClientActors became real was take the stick-figure animations out of my original NPC scripts, and throw in ClientActors in their place. Not bad, but the code wasn't great, so I put it aside and started from scratch.

Video 2 - I quickly learned that ClientActors don't do anything unless you tell them to do it. That includes taking damage. Here is some preliminary hit detection and ragdolling.

Video 3 - After that, I started working on syncing the ClientActors using WorldNetworkObjects, which are really powerful objects if used properly. By pairing each CA to a WNO (marked by the blue circle) and wrapping them in a custom NPC class, I was able to make the CAs follow the WNOs across the ground. This is before I finished my terrain map, so there is no pathfinding involved here.

Video 4 - Watch how he runs faster to catch up to the WNO if he is too far from it.

Video 5 - After syncing movement, I moved on to pathfinding. I had a little trouble at first. This is cell 18x16 that I mentioned earlier.

Video 6 - I had enough of that, so I went back to revisit ragdolling. Turns out if you set the animation state to SDead at the right times, you can get a slow ragdoll...

Video 7 - ... or a weirdly immediate ragdoll.

Video 8 - Like everything else they don't do, CAs do not respond to vehicle collisions unless you tell them they've been hit by a vehicle. Fear the ice cream truck.

Video 9 - Speaking of vehicles, I wanted to see how possible it is to make CAs fly a plane. I jury rigged my autopilot script so that it would tell the CA how to fly, and it worked!

Video 10 - Back to syncing positions. This is an example of a situation where you have to decide which client to use to sync the CA position. Shown here is a LocalPlayer sending position updates to the WNO if it is next to (or inside of) the LocalPlayer's vehicle. Notice how the blue circle moves as it receives the updated position.

Video 11 - Back to pathfinding, which I had sorted it out by now. This is an example of 4-direction movement on a 1 meter grid, with the NPC detecting and pathing around a vehicle. To facilitate this, I wrote a script that keeps track of vehicle positions so that NPCs do not have to check every vehicle, only nearby ones.

Video 12 - Just working on hit detection here, with a scatter shot from a shotgun.

Video 13 - NPCs pathing around. It almost looks natural, except for the fact that they are wandering aimlessly.

Video 14 - Here is another example of an NPC pathing around an occupied vehicle, this time with 8-direction movement and a 2 meter grid.

Video 15 - This is what happens if the NPC does not check for vehicles, and the CA runs into an unoccupied vehicle.

Video 16 - This is what happens if the NPC does not check for vehicles, and the CA runs into an occupied vehicle. RIP.

Video 17 - Took a quick look at following AI. He seems a bit drunk.

Video 18 - How about melee damage?

Video 19 - Here is a terminator showing off different hit reactions based on where he is hit.

Video 20 - His buddies were not so lucky, but at least we know that different ragdolls are possible.

Video 21 - EXPLOSIONS. Had to simulate a rocket for this one.

Video 22 - Had to simulate grenades, too. Here's an example of what is not supposed to happen.

Video 23 - Blowing up an unoccupied vehicle with a rocket launcher. Just Cause.

Video 24 - Oh. This is how I built my climate map. Lols.

Video 25 - Lastly, here is a video of civilian NPCs on Jman100's demo server. Notice the desert clothing in the desert. Also notice that they (try to) run away.

----

That's basically where I am at the moment. I took a break from working on NPCs in order to work on some other things. When I come back to this, the I really need to work on optimizing their logic before I go any further. You might notice in the last video that they are a bit laggy and derpy, mostly because they are pretty good resource hogs at the moment, also because the pathing logic can use some work. Most of this is intended to be a proof of concept rather than a practical solution, for now, at least.
« Last Edit: September 22, 2015, 06:37:42 am by SinisterRectus »

harisalipk

  • Donator
  • Jr. Member
  • *****
  • Posts: 78
    • View Profile
Re: Basic NPC AI
« Reply #41 on: September 22, 2015, 07:51:01 pm »
Mhm mhm, I know some of these words...

Seriously though, thanks for taking the time to post your progress in such great detail. It's very interesting to see what goes on behind the scenes and how much effort is required to create new features for the mod.

SinisterRectus

  • JC2-MP Betatester
  • Sr. Member
  • *****
  • Posts: 451
    • View Profile
Re: Basic NPC AI
« Reply #42 on: November 15, 2015, 05:06:32 am »
I took a break from scripting on-foot AI because, frankly, coordinating the movement of dozens or hundreds of NPCs on a single server is hard work.

Never the less, today I put together something a little more simple: Air traffic! It's mostly modeled after single-player air traffic, where planes just fly across the sky in a straight line. No WorldNetworkObjects here. Just Vehicles streaming in and out, with ClientActor pilots. I used a modified version of my autopilot to control the planes on the client, while the server manages the positions and velocities.

Video here.
« Last Edit: November 15, 2015, 06:50:15 am by SinisterRectus »

JasonMRC

  • Donator
  • Hero Member
  • *****
  • Posts: 601
    • View Profile
Re: Basic NPC AI
« Reply #43 on: November 15, 2015, 05:14:01 am »
Very nice. I do wonder how you scripted ripping the pilot out, or does the game automatically throw out client actors?

SinisterRectus

  • JC2-MP Betatester
  • Sr. Member
  • *****
  • Posts: 451
    • View Profile
Re: Basic NPC AI
« Reply #44 on: November 15, 2015, 05:24:55 am »
Interestingly enough, it's handled automatically by the game. You have to manually keep track of the vehicle's occupancy, though.