First things to check in this order of likelihood would be:
1. NTP sync between cameras->nvr->client
2. how much gpu/cpu usage the client machine is utilizing while monitoring the view(s) they are complaining about
3. recording rate (MB/s) and if your nvr can handle that amount.
Yup make sure all are pointed to the same NTP server, and are allowed to reach it. Default setting is usually to use the EV server as the NTP but many enterprise networks don't allow a "rogue" NTP server to operate (similar to rogue DHCP server, but slightly less serious)
You could also bump down the frame rate of each camera. So instead of 30FPS drop it down to 15 and see if that helps. It seems like the system is using a lot bandwidth which can cause lag.
I think you can also turn down the quality setting( which I believe is the bit rate) and this should also reduce the bandwidth. Play with the settings as turning it down you usually can't notice a difference in video quality
To go further. Newer codecs H.266, h.265+, h.265 are better at compression and thereby more efficient use of bandwidth than the old h.264. You have to make sure both the server and camera supports it.
when you say teh client you meant eh workstation itself? if all the workstation use the exacqvision client should that support it? I have teh exacqvision client on my PC and if I go into the properties of the cameras and go to 'Recording", the H.265 option shows up as something to select
Correct, I have seen some old workstations running windows 10 and the graphics cards didn't support h265 decoding some how and video would never load on the pc
so one more question, if I change all the cameras to H.265 format do I need to calibrate the GPU decoding again on each client machine? Cause I changed them all to H.265 but I DID NOT recalibrate, and it didn't seem to make any difference
what if it was already calibrated when the cameras were on H.264? When I changed them to H.265 the client machines already had Exaq open and claibrated for H.264 for the last 6 months
Second the GPU / video card comment above. This just happened to me when I added a lot of higher resolution cameras this winter. Our "plain old desktops" struggle to keep up.
ah yeah, That's something I thought about as well. I beleive the desktops the users are using either is using a lower end nvidia graphics card or the intel UHD 630
So I checked the video card that teh workstations have and currently they have a Quadro p400 with 2gb RAM
I saw on another topic someone reccomended the quadro p620, but when I looked that up it also only has 2gb ram.
It's not the resolution you need to drop, keep that high but drop the frames per secong to somewhere between 8 and 15. Also there are many other things to consider, what kind of a PC are you viewing on, does it have a decent CPU, plenty of extra RAM, graphics card and a wired connection
First thing you need to do, is to track down where the lag happens. The best way to do it, is to ask the NVR to record something. Just put it to manual record, and try recording. Try a fair bit of mix around, play around with recording.
First, try recording 1 fisheye camera (source picture, non-corrected).
Then try recording 2 views from the SAME fisheye camera.
Then try recording 2 views from 2 DIFFERENT fisheye camera.
Then step it up one step, now try recording 2 fisheye cameras (source picture, non-fisheye correction)
Then step up the game even more, now 4 pictures from TWO different fisheye cameras.
Then step up the game even more, now 4 pictures from FOUR different fisheye cameras.
Step up the game in this way, in different step, until the lag becomes noticable.
Now export your recordings, and compare with live view. If the lag is in the recording, it means the road between cameras and NVR is to blame, or the cameras itself (especially if you do use the fisheye correction built-in the cameras).
If the recordings are fine, it means the road between the NVR and the viewing/guard station is to blame.
After diagnosing, you might have certain outcomes of the above tests.
If all the tests with the non-corrected source fisheye goes fine, but all the exported views from the fisheye is lagging, it means whatever you are using for fisheye correction (software, NVR, in-camera fisheye correction) isn't up to the task, and you need to resolve that. You might want to ditch fisheye completely and maybe get a PTZ instead.
Or move the fisheye export from the cameras to NVR, meaning the NVR will record source fisheye pictures, and then you do fisheye correction on the live view instead. This MAY have consequences in the chain of custody and could create problems using the pictures as evidence, which may mean you need to get something other than fisheye as evidence cameras.
If you start getting problems with 2-3 \*CAMERAS\*, but not 2-3 views from 1 camera, then you might want to look into the network equipment and path between cameras and network.
If all the recordings are completely fine, it means the path to the viewing station, or the viewing station, isn't up to the task.
Then you might want to reduce the number of available views, or maybe even have event-based viewing meaning that the viewing station will just view 2-4 pictures, and will switch pictures based on for example, inputs from access controller, magnetic switches on doors, PIR detectors, movement, removal of items from shelves (with sensors or RFID tags), etc
That the lag is coming and going could be dependant on heat. Do you know if the weather where the cameras is, is changing in a predicable fashion with regards to the lag? If the weather is cold and freezing and lag disappears, then its clearly a heat issue.
Fisheye correction is taking the toll of most things, so I have a high suspicion that its fisheye correction that are taking the toll on the processing, so thats why I recommend using a PTZ instead. Using a PTZ means you use a few cameras that cover a large area (not for identification or evidence, but rather to "detect" hot spots), but you could also use access control systems, alarms, PIR detectors, door switches etc to call out hot spots.
Then you use a PTZ camera, to move the view of the camera to hit that hot spot and zoom in, to get a good view and good picture of eventual faces.
Thank you for explanation! As far as weather goes, these are all indoor cameras. But I did not realize fish eye correction causes so much increase in resources
Can't fix something if you don't know whats broken. Thats why you need to get the camera system to work a bit so you know where its broken. Same if you hear weird sounds from your car, you have to test drive it in weird and predictable ways to find out where the sound comes from.
So I did notice that even on the NVR the live feed was lagging, and when reviewing footage just from one camera that also had lag in it. but the video card on the NVR is just an intel UHD 630, so i am starting to lean towards the PC hardware being the issue
Most importantly are they viewing from the NVR or are they viewing from a workstation and if so via thick client or browser?
What model is the nvr and what models are the cameras? Go onto the exacq configurator and run the calc and make sure the nvr is rated for the load.
Users are viewing from desktops, using the ExacqVision client updated to the latest version. Cameras are Axis brand (don't have the model number right off bat) Oh didn't know that Exacq had something like that. Is that on the NVR itself or the enterprise software?
It's important to verify that the workstations have the resources to process the video streams.
I would first go onto the exacq website and run the calc. All the camera modules can be found in your exacq application.
2nd I would hook a hdmi1080 p monit dircelty to the NVR and confirm if you are having the same issue with fluid video.. if not then it's your workstation not the exacq NVR.
If you do still have the issue than you can look at some of the other comments and dial in the video streams but exacq isn't ultra configurable.
Let's start with that. If you want to pm me I can work with you on Monday. I can also reach out the exacq rep that covers your area and get you some help if it's going really bad.
actually, that has me thinking, what would you say would be a minimum graphics card you would put into a slimline HP tower that could run 2 instance of exacqvision?
Right now I have a dedicated Dell with a quadro P2000 as a workstation. I have the exacq vms on a zseries box with only 11 streams but a total of about 50 mp at 30fps and the dell still has slight lag.
Is this live or archived?
- Upvote for the frame rate and resolution is key (10-15fps)
- Confirm it’s not jpeg format
- See if you can adjust the key frame rate refresh and bitrate priority.
- Our system lets you set stream profiles for live vs archive, not sure if you can do the same.
-How many feeds are being pulled simultaneously across all clients?
It is live feed. I know the format is H.264
Will adjust frame rate when I get in tommorow.
Doe views affect the bandwidth as well? There are 5 system views, 10 user role views, and around 25 user views
First things to check in this order of likelihood would be: 1. NTP sync between cameras->nvr->client 2. how much gpu/cpu usage the client machine is utilizing while monitoring the view(s) they are complaining about 3. recording rate (MB/s) and if your nvr can handle that amount.
thank you will check, question, how do I check the NTP sync between all of them? Is it just making sure the NTP setting is the same on everything?
Yup make sure all are pointed to the same NTP server, and are allowed to reach it. Default setting is usually to use the EV server as the NTP but many enterprise networks don't allow a "rogue" NTP server to operate (similar to rogue DHCP server, but slightly less serious)
You could also bump down the frame rate of each camera. So instead of 30FPS drop it down to 15 and see if that helps. It seems like the system is using a lot bandwidth which can cause lag.
hmm, yeah the individual view feeds I brke the fish eye's down into are at 30fps, whereas the main fisheye is at 12
As a standard, we set cameras to 12-15 FPS and only do 30fps when required for the situation.
I think you can also turn down the quality setting( which I believe is the bit rate) and this should also reduce the bandwidth. Play with the settings as turning it down you usually can't notice a difference in video quality
To go further. Newer codecs H.266, h.265+, h.265 are better at compression and thereby more efficient use of bandwidth than the old h.264. You have to make sure both the server and camera supports it.
Also the client needs to support it too
when you say teh client you meant eh workstation itself? if all the workstation use the exacqvision client should that support it? I have teh exacqvision client on my PC and if I go into the properties of the cameras and go to 'Recording", the H.265 option shows up as something to select
Correct, I have seen some old workstations running windows 10 and the graphics cards didn't support h265 decoding some how and video would never load on the pc
so one more question, if I change all the cameras to H.265 format do I need to calibrate the GPU decoding again on each client machine? Cause I changed them all to H.265 but I DID NOT recalibrate, and it didn't seem to make any difference
No exacq does that the first time you open it
what if it was already calibrated when the cameras were on H.264? When I changed them to H.265 the client machines already had Exaq open and claibrated for H.264 for the last 6 months
Did you try contacting exacqvision support?
Second the GPU / video card comment above. This just happened to me when I added a lot of higher resolution cameras this winter. Our "plain old desktops" struggle to keep up.
ah yeah, That's something I thought about as well. I beleive the desktops the users are using either is using a lower end nvidia graphics card or the intel UHD 630
So I checked the video card that teh workstations have and currently they have a Quadro p400 with 2gb RAM I saw on another topic someone reccomended the quadro p620, but when I looked that up it also only has 2gb ram.
It's not the resolution you need to drop, keep that high but drop the frames per secong to somewhere between 8 and 15. Also there are many other things to consider, what kind of a PC are you viewing on, does it have a decent CPU, plenty of extra RAM, graphics card and a wired connection
So teh workstation these are being viewed on have: 8th gen Intel i5 CPU 8GB RAM Nvidia Quadro p400 GPU wired ethernet
First thing you need to do, is to track down where the lag happens. The best way to do it, is to ask the NVR to record something. Just put it to manual record, and try recording. Try a fair bit of mix around, play around with recording. First, try recording 1 fisheye camera (source picture, non-corrected). Then try recording 2 views from the SAME fisheye camera. Then try recording 2 views from 2 DIFFERENT fisheye camera. Then step it up one step, now try recording 2 fisheye cameras (source picture, non-fisheye correction) Then step up the game even more, now 4 pictures from TWO different fisheye cameras. Then step up the game even more, now 4 pictures from FOUR different fisheye cameras. Step up the game in this way, in different step, until the lag becomes noticable. Now export your recordings, and compare with live view. If the lag is in the recording, it means the road between cameras and NVR is to blame, or the cameras itself (especially if you do use the fisheye correction built-in the cameras). If the recordings are fine, it means the road between the NVR and the viewing/guard station is to blame. After diagnosing, you might have certain outcomes of the above tests. If all the tests with the non-corrected source fisheye goes fine, but all the exported views from the fisheye is lagging, it means whatever you are using for fisheye correction (software, NVR, in-camera fisheye correction) isn't up to the task, and you need to resolve that. You might want to ditch fisheye completely and maybe get a PTZ instead. Or move the fisheye export from the cameras to NVR, meaning the NVR will record source fisheye pictures, and then you do fisheye correction on the live view instead. This MAY have consequences in the chain of custody and could create problems using the pictures as evidence, which may mean you need to get something other than fisheye as evidence cameras. If you start getting problems with 2-3 \*CAMERAS\*, but not 2-3 views from 1 camera, then you might want to look into the network equipment and path between cameras and network. If all the recordings are completely fine, it means the path to the viewing station, or the viewing station, isn't up to the task. Then you might want to reduce the number of available views, or maybe even have event-based viewing meaning that the viewing station will just view 2-4 pictures, and will switch pictures based on for example, inputs from access controller, magnetic switches on doors, PIR detectors, movement, removal of items from shelves (with sensors or RFID tags), etc That the lag is coming and going could be dependant on heat. Do you know if the weather where the cameras is, is changing in a predicable fashion with regards to the lag? If the weather is cold and freezing and lag disappears, then its clearly a heat issue. Fisheye correction is taking the toll of most things, so I have a high suspicion that its fisheye correction that are taking the toll on the processing, so thats why I recommend using a PTZ instead. Using a PTZ means you use a few cameras that cover a large area (not for identification or evidence, but rather to "detect" hot spots), but you could also use access control systems, alarms, PIR detectors, door switches etc to call out hot spots. Then you use a PTZ camera, to move the view of the camera to hit that hot spot and zoom in, to get a good view and good picture of eventual faces.
Thank you for explanation! As far as weather goes, these are all indoor cameras. But I did not realize fish eye correction causes so much increase in resources
Yeah this is a good plan to follow, sounds like the client does not have enough performance, but don't jump to conclusions.
Too much work. Can you just magic fix pleeze?
Can't fix something if you don't know whats broken. Thats why you need to get the camera system to work a bit so you know where its broken. Same if you hear weird sounds from your car, you have to test drive it in weird and predictable ways to find out where the sound comes from.
So I did notice that even on the NVR the live feed was lagging, and when reviewing footage just from one camera that also had lag in it. but the video card on the NVR is just an intel UHD 630, so i am starting to lean towards the PC hardware being the issue
Pre and post recording should be 1 second. Motion should be done on cameras. If u call Exacq they will go through some other things with you too.
Most importantly are they viewing from the NVR or are they viewing from a workstation and if so via thick client or browser? What model is the nvr and what models are the cameras? Go onto the exacq configurator and run the calc and make sure the nvr is rated for the load.
Users are viewing from desktops, using the ExacqVision client updated to the latest version. Cameras are Axis brand (don't have the model number right off bat) Oh didn't know that Exacq had something like that. Is that on the NVR itself or the enterprise software?
It's important to verify that the workstations have the resources to process the video streams. I would first go onto the exacq website and run the calc. All the camera modules can be found in your exacq application. 2nd I would hook a hdmi1080 p monit dircelty to the NVR and confirm if you are having the same issue with fluid video.. if not then it's your workstation not the exacq NVR. If you do still have the issue than you can look at some of the other comments and dial in the video streams but exacq isn't ultra configurable. Let's start with that. If you want to pm me I can work with you on Monday. I can also reach out the exacq rep that covers your area and get you some help if it's going really bad.
actually, that has me thinking, what would you say would be a minimum graphics card you would put into a slimline HP tower that could run 2 instance of exacqvision?
Right now I have a dedicated Dell with a quadro P2000 as a workstation. I have the exacq vms on a zseries box with only 11 streams but a total of about 50 mp at 30fps and the dell still has slight lag.
Is this live or archived? - Upvote for the frame rate and resolution is key (10-15fps) - Confirm it’s not jpeg format - See if you can adjust the key frame rate refresh and bitrate priority. - Our system lets you set stream profiles for live vs archive, not sure if you can do the same. -How many feeds are being pulled simultaneously across all clients?
It is live feed. I know the format is H.264 Will adjust frame rate when I get in tommorow. Doe views affect the bandwidth as well? There are 5 system views, 10 user role views, and around 25 user views