Ranger Firing Range - Testing

Eorzea Time
 
 
 
Language: JP EN FR DE
users online
Forum » FFXI » Jobs » Ranger » Ranger Firing Range - Testing
Ranger Firing Range - Testing
 Leviathan.Comeatmebro
Online
Server: Leviathan
Game: FFXI
user: Rairin
Posts: 6052
By Leviathan.Comeatmebro 2017-02-14 12:32:03
Link | Quote | Reply
 
Don't really see the benefit to sticking to 600, given the high probability it's a model similar to conserve mp it seems pretty important to get the best resolution possible. I'm not using snapshot, so as far as nailing down -just- rapid shot, I don't see the relevance.
[+]
 Leviathan.Comeatmebro
Online
Server: Leviathan
Game: FFXI
user: Rairin
Posts: 6052
By Leviathan.Comeatmebro 2017-02-14 12:32:47
Link | Quote | Reply
 
I'm not opposed to doing multiple tests or anything though, half my characters are tied up doing RDM job points and I'm just using the spares to test. Maybe relic bow instead of gun and do the baseline on SAM. Don't expect any results until late tonight or sometime tomorrow though, gotta actually level RNG and SAM and farm a relic bow along with valentines stuff.
 Lakshmi.Byrth
VIP
Offline
Server: Lakshmi
Game: FFXI
user: Byrthnoth
Posts: 6137
By Lakshmi.Byrth 2017-02-14 13:08:15
Link | Quote | Reply
 
https://www.bluegartr.com/threads/116145-Ranged-Delay-Snapshot-and-Stuff?p=5737468&viewfull=1#post5737468

Rapid Shot can reduce delay by up to 50%, at least.

I kinda bet throwing rings aren't affected by Snapshot, but they do appear to be affected by Rapid Shot so they could perhaps be used to isolate it.
 Leviathan.Comeatmebro
Online
Server: Leviathan
Game: FFXI
user: Rairin
Posts: 6052
By Leviathan.Comeatmebro 2017-02-14 13:14:33
Link | Quote | Reply
 
What would you recommend using to provide the most additional data in the least testing?
 Lakshmi.Byrth
VIP
Offline
Server: Lakshmi
Game: FFXI
user: Byrthnoth
Posts: 6137
By Lakshmi.Byrth 2017-02-14 13:17:35
Link | Quote | Reply
 
Use Self Bow, capped Snapshot, and as much rapid shot as you can get on THF to do a few hundred shots.

Your Self Bow sample was really homogenous, so we should (I'd hope) be able to identify rapid shot procs. If you're using a known amount of Rapid Shot, then we'll have something to check against. Once we have a population of shots where Rapid Shot procced and a population where they didn't, we can divide the averages and get the delay reduction due to Rapid Shot at capped Snapshot.
Offline
By Verda 2017-02-14 13:18:01
Link | Quote | Reply
 
So I graphed Comeatmebro's data:



Orange = Actual Data set
Yellow = Slope of line calc'd from the 210 delay Bow and 360 delay Bow
Green = Slope of the line calc'd with 540 and 600 delay weapons
Brown = Line of best fit on the data

The slope isn't very smooth so that worries me.

What I do know is that this is how far each point is, in ms, from the line of best fit:
Code
Velocity Bow      -163.91
Shortbow          11.59
Self Bow          -115.11
Lamian Kaman -1   85.19
Pirate's Gun      -28.61


None of them are even to 200ms so sample seems ok. But it still is hard to say what the slope ends up being, I mean for using Pirate's Gun and Lamian kaman -1, the value at 0 delay would be about 3.1 seconds but using the best fit line it ends up being about 2.015 seconds and using the lower bound line it's 1.6 seconds.

That's a huge gap of up to 1.5 seconds. The outliers in this data set seem to be Pirate's Gun and Self Bow which don't really follow the established line of the other 3 very closely at all. Since we have 5 though it's hard to know if they're the outliers or the other 3 are.

Anyway, it feels pretty hard to come to a conclusion with this but it's still really useful, I have to get a lot of work done for work but I'll revisit here my first chance and let this stuff mull over in my mind the rest of the day.

Edit:
Also if you want my excel sheet I made the graph it's here:
http://www.mediafire.com/file/hkwwmzamni6y0vj/rng_delay_vs_real2.xlsx
[+]
 Leviathan.Comeatmebro
Online
Server: Leviathan
Game: FFXI
user: Rairin
Posts: 6052
By Leviathan.Comeatmebro 2017-02-14 13:19:10
Link | Quote | Reply
 
Don't think the differences will be too low to properly identify it? I thought I read a theory that it was 1/16 to 8/16 delay reduction like conserve mp, wouldn't change the number of packets consistantly with that little delay.
 Lakshmi.Byrth
VIP
Offline
Server: Lakshmi
Game: FFXI
user: Byrthnoth
Posts: 6137
By Lakshmi.Byrth 2017-02-14 13:21:44
Link | Quote | Reply
 
I can't be sure, but Mojo's data would indicate it might be possible.

The other option is to use a known amount of Rapid Shot while keeping capped Snapshot, repeat your whole curve, and just compare the slopes.

For instance,
0% Rapid Shot Slope: 2.53
X% Rapid Shot Slope: Y
Y = 2.53*(1-X) + X*z

In this case, Y and X would be constants we find from your data and z would be the average value of Rapid Shot at capped Snapshot.
 Leviathan.Comeatmebro
Online
Server: Leviathan
Game: FFXI
user: Rairin
Posts: 6052
By Leviathan.Comeatmebro 2017-02-14 13:25:23
Link | Quote | Reply
 
I was thinking something like:

250 shots with relic bow on SAM(no snap or rapid)

250 shots with relic bow on SAM(40 snap, no rapid.. could achieve this with 10% snap cape and flurry2)

250 shots with relic bow on RNG(no snap, 100+ rapid)

250 shots with relic bow on rng(40 snap, 100+ rapid.. could achieve this with 10% snap cape and flurry2)
 Lakshmi.Byrth
VIP
Offline
Server: Lakshmi
Game: FFXI
user: Byrthnoth
Posts: 6137
By Lakshmi.Byrth 2017-02-14 13:27:04
Link | Quote | Reply
 
That would work, but I don't think you care about the value of Rapid Shot at 40% Snapshot.
 Leviathan.Comeatmebro
Online
Server: Leviathan
Game: FFXI
user: Rairin
Posts: 6052
By Leviathan.Comeatmebro 2017-02-14 13:27:25
Link | Quote | Reply
 
If you don't think that'd provide any additional information, obviously rather do the shorter test with self bow. Just seems to me it'd be hard to pick out the differences given a 1/16 rapid shot proc probably isn't enough to push most shots a packet interval away.

Nah, but it'd show how rapid shot interacts with snapshot delay-wise. From that you should be able to get pretty close to results at capped snapshot. I guess ideal test for actual benefit would be a real set(max possible rapid shot w/ capped snapshot with and without velocity).
 Lakshmi.Byrth
VIP
Offline
Server: Lakshmi
Game: FFXI
user: Byrthnoth
Posts: 6137
By Lakshmi.Byrth 2017-02-14 13:28:53
Link | Quote | Reply
 
Even if it doesn't work, it would be a starting point for doing the entire curve that way.


And yeah, if the point of this testing is to find the delay you actually experience, the best test would be to . . . shoot with the set you actually use.
 Fenrir.Snaps
Offline
Server: Fenrir
Game: FFXI
user: Mojopojo
Posts: 1139
By Fenrir.Snaps 2017-02-14 13:31:52
Link | Quote | Reply
 
Lakshmi.Byrth said: »
https://www.bluegartr.com/threads/116145-Ranged-Delay-Snapshot-and-Stuff?p=5737468&viewfull=1#post5737468

Rapid Shot can reduce delay by up to 50%, at least.

I kinda bet throwing rings aren't affected by Snapshot, but they do appear to be affected by Rapid Shot so they could perhaps be used to isolate it.

I am seeing 11 peaks of observed delay in your data, not 9 (as would be expected with a n = 1 through 8 n/16 model).

Also there's the possibility that there is some kind of compound delay where each portion is either affected by RS or SS/VS. Probably not true since Byrth's data indicates otherwise.
 Lakshmi.Byrth
VIP
Offline
Server: Lakshmi
Game: FFXI
user: Byrthnoth
Posts: 6137
By Lakshmi.Byrth 2017-02-14 13:50:05
Link | Quote | Reply
 
Those peaks are the average incoming packet interval multiples. Really what it tells us is that whatever Rapid Shot does causes a pretty flat distribution of delay reduction.

In itself, if we accept the minimum and we accept that the delay reduction amount is relatively flat, we're really just debating the difference between like 75% average delay on procs (0-50% delay reduction with a flat probability distribution) and some model that doesn't start from 0% or is discretized strangely to give a slightly different result.

Unless at extremely low delays (thanks to Snapshot), Rapid Shot's impact on delay changes. These extremely low delays are really the only delays relevant to /ra jobs, so that might as well be the target.
 Fenrir.Snaps
Offline
Server: Fenrir
Game: FFXI
user: Mojopojo
Posts: 1139
By Fenrir.Snaps 2017-02-14 13:53:12
Link | Quote | Reply
 
Ah, yeah that makes sense. So it could be n/8 still. I suppose the simplest method would be to collect a lot of RS only data (with a known RS rate, so no traits) and see if the mean delay reduction matches the n/8 expectation.
 Fenrir.Snaps
Offline
Server: Fenrir
Game: FFXI
user: Mojopojo
Posts: 1139
By Fenrir.Snaps 2017-02-14 15:20:53
Link | Quote | Reply
 
Verda said: »
So I graphed Comeatmebro's data:



Orange = Actual Data set
Yellow = Slope of line calc'd from the 210 delay Bow and 360 delay Bow
Green = Slope of the line calc'd with 540 and 600 delay weapons
Brown = Line of best fit on the data

The slope isn't very smooth so that worries me.

What I do know is that this is how far each point is, in ms, from the line of best fit:
Code
Velocity Bow      -163.91
Shortbow          11.59
Self Bow          -115.11
Lamian Kaman -1   85.19
Pirate's Gun      -28.61


None of them are even to 200ms so sample seems ok. But it still is hard to say what the slope ends up being, I mean for using Pirate's Gun and Lamian kaman -1, the value at 0 delay would be about 3.1 seconds but using the best fit line it ends up being about 2.015 seconds and using the lower bound line it's 1.6 seconds.

That's a huge gap of up to 1.5 seconds. The outliers in this data set seem to be Pirate's Gun and Self Bow which don't really follow the established line of the other 3 very closely at all. Since we have 5 though it's hard to know if they're the outliers or the other 3 are.

Anyway, it feels pretty hard to come to a conclusion with this but it's still really useful, I have to get a lot of work done for work but I'll revisit here my first chance and let this stuff mull over in my mind the rest of the day.

Edit:
Also if you want my excel sheet I made the graph it's here:
http://www.mediafire.com/file/hkwwmzamni6y0vj/rng_delay_vs_real2.xlsx

In the 600 delay tests I posted, you could see that there was some variation of .03~ seconds even though I had 200+ samples of each. I'll go back over the data and post the standard deviation of each weapon so we can consider the confidence interval on them. That said, both data sets using more data across a variety of delays seemed to point to the same number (1.8).
 Leviathan.Comeatmebro
Online
Server: Leviathan
Game: FFXI
user: Rairin
Posts: 6052
By Leviathan.Comeatmebro 2017-02-14 20:31:14
Link | Quote | Reply
 
RNG/DNC
85% rapid shot gear, 5 merits, assume trait caps. No snapshot.
Kustawi+1/kustawi/relic bow/stone arrow
pursuer's beret[A]/loricate torque+1/etiolation earring/sanare earring
Arcadian Jerkin +1/Mrigavyadha Gloves/defending ring/shneddick ring
solemnity cape/yemaya belt/pursuer's pants[A]/pursuer's gaiters[A]
Indi-Regen, Geo-Refresh, no flurry.

Total shots: 1000
Fastest shot: 5711ms
Slowest shot: 10235ms
Mean shot: 7910ms

Probably not useful to anyone until I get a baseline, but here it is anyway:
https://www.dropbox.com/s/jzdshv1t7zqzi0p/20170214_191029.log?dl=0
and raw:
https://www.dropbox.com/s/42yg1j60wdso6hq/20170214_191029.raw?dl=0

Tests for tomorrow(need to level SAM):
SAM/WAR 0 snapshot relic bow/stone arrow 250 shots
SAM/WAR 10 snapshot+flurry2 relic bow/stone arrow 250 shots
RNG/DNC 10 snapshot+flurry2+above setup relic bow/stone arrow 1000 shots
Probably not the most practical of tests, but like I said I have the time atm and it's all automated so it isn't much work. I still think this is the best way to accurately nail down what rapid shot is intended to do, even if packet structure causes less effective results in practice.

THF/WHM 60 snapshot flurry1 32 rapid shot:
velocity bow+1/sleep bolt 250 shots
shortbow/stone arrow 250 shots
self bow/stone arrow 250 shots
lamian kaman-1/stone arrow 250 shots
pirate's gun/bronze bullet 250 shots
aureole 250 shots
[+]
 Fenrir.Snaps
Offline
Server: Fenrir
Game: FFXI
user: Mojopojo
Posts: 1139
By Fenrir.Snaps 2017-02-15 02:28:09
Link | Quote | Reply
 
Some Rapid Shot data. 0 Snapshot/Velocity Shot was used with a Shortbow. RSII = Rapid Shot II + 5 merit points (yes I did remove my Snapshot merits for this.)

Code
RS   n   Delta         sDelta        Expected
rs00 594 4.87222934233 3.26262962963 4.879819298
rs25 700 4.67080400572 3.08963571429 4.66326950361
rs50 715 4.42383893557 2.85989090909 4.44671970922
rsII 580 4.53685319516 2.96841206897 4.55066361053
Delta LSR (delta between each ranged attack begin packet)

m = -.008967808135
b = 4.879819298
R^2 = .9965734553

sDelta LSR (delta between each ranged attack begin and ranged attack finish pair)

m = -.008054774411
b = 3.272088111
R^2 = .9934247495

Using the data for RSII and the Delta LSR yields RSII = 38.2 (33 + 5)
Using the data for RSII and the dDelta LSR yields RSII = 37.7 (33 + 5)

Rapid Shot II is probably 33 then (unless Velocity Shot gifts apply without Velocity Shot being active, but I don't think they do.)

I added an expected column to the data which follows the JP wiki formula using a 1.8 second static delay + cooldown (seems reasonably accurate.)
 Lakshmi.Byrth
VIP
Offline
Server: Lakshmi
Game: FFXI
user: Byrthnoth
Posts: 6137
By Lakshmi.Byrth 2017-02-15 05:31:12
Link | Quote | Reply
 
Re: Mojo
If you plot the sDelta and look at the slope, it predicts that at 0% Snapshot we'd get a 24% reduction in delay with 100% Rapid Shot. That's consistent with a 25% reduction in delay (as if it was a flat distribution of 0-50% reduction).
Offline
By Verda 2017-02-15 10:01:39
Link | Quote | Reply
 
Fenrir.Snaps said: »
In the 600 delay tests I posted, you could see that there was some variation of .03~ seconds even though I had 200+ samples of each. I'll go back over the data and post the standard deviation of each weapon so we can consider the confidence interval on them. That said, both data sets using more data across a variety of delays seemed to point to the same number (1.8).

It doesn't point to the same number though. Here is your data with the line of best fit, and comeatmebro's data with the line of best fit:


Yours at zero goes to 1807 and his at zero goes to 2015.

The line for yours is y=1.25x + 1803 (brown)
The line for his is y=.233333x + 3125 (green)
Your data set is orange and comeatmebro's is yellow.

As stated in my last post I don't really see how reducing delay to zero is going to give an accurate result, which is why I showed the lower and upper bound lines having over 1.5s difference in the first place on his data.

We definitely can't just say it's 1.8 and they agree, because they don't, and using a method of setting delay = 0 isn't really dependable unless the data has no noise, which it does.

What you can see from this is, velocity shot + rapid shot you have even tho Comeatmebro's method shoots faster your shots for the given delay were all much faster so if it was still up in the air for anybody, which I hope it wasn't, then this should demonstrate that velocity and rapid or at least one of them break snapshot caps.

Fenrir.Snaps said: »
It is definitely a big kick to the balls to the perceived usefulness of lower delay weapons and ranger in general :(
Going to disagree here as well. As shown in my first reply to your data, low delay weapons if anything can have more tp/s while still shooting faster while still maintaining 3 hit sets. The tradeoff is lower base damage unless you're using a magic ws, in which case base damage won't matter for ws damage.

Also as stated in a prior post, the delay denominator is a lot different for ranged than melee.

The effect you *can* say the effect at least the noise has, is it makes delay have less wait in attack speed (but still matter for tp calc).

On Melee you'd expect something with half the delay to attack 2x as often. On ranged, going by bro's data and using the line of best fit:
600 delay has 3293 cycle time and 300 delay has 2654 cycle time that is: 1.24 or 24% increase in attack speed using half the delay of weapon. So 76% of the benefit you would expect from lower delay is gone, but then you have to consider: 1) almost double the divisor for delay on ranged attacks 2) ammo delay for free tp (which favors lower delay weapons) 3) arcadian beret ammo for free (which favors lower delay weapons) 4) the nature of rng in practice and 3 hit sets

All of which I outlined in prior posts.
Offline
By Verda 2017-02-15 10:15:58
Link | Quote | Reply
 
As far as how to make sense of this data, it seems we need to just put a lot of additional average noise. We got between 1.8 and a bit over 2s. 1 sec of that is already accounted for on the spreadsheet. The other .9 seconds however I just added. It makes the end dps numbers seem more reasonable. I also put it up front, and easily editable as Comeatmebro suggested. I called it Avg+ Delay Noise unless we can think up a better term for it.

http://www.mediafire.com/file/59a44092sfk3zdg/DPS_Calculator_-_Rng-custom6.69.xlsx

Review appreciated. I used a method of making 2 weaponskill sheets, one that always recycle procs and one that doesn't, to obtain a weighted average of shots. I couldn't fix the actual ws sheet and not having it in the calcs at all was really hurting some setups. It's not perfect so still working on it, but it doesn't reduce the rounds till ws by more than if recycle proc'd 100% of the time like it was doing when byrth and I first started working on it and it also doesn't just not include arcadian recycle procs either like it was doing before.

I'm still not happy with this method tho, I think we should break every shot down into it's components to identify the noise.

Also, really appreciate the work and input of everyone this is fun to me.
 Fenrir.Snaps
Offline
Server: Fenrir
Game: FFXI
user: Mojopojo
Posts: 1139
By Fenrir.Snaps 2017-02-15 10:38:07
Link | Quote | Reply
 
I think you need to redo your math. The LSR for both data sets point to 1.8 seconds. I also used his method of injecting a ranged attack on ever packet so there was no difference other than Windower vs Ashita.

I'll do more follow up after work.
Offline
By Verda 2017-02-15 12:17:54
Link | Quote | Reply
 
I graphed it and shared the spreadsheet just so you couldn't use that as an argument. The only way comeatmebro's data gets close to 1.8 (well i guess you could argue a bit over 2 is close to 1.8) is if you only use the lower bound which as was shown can really skew the slope.

The (x,y) points for your data are:
130,1967
360,2276
450,2365
540,2438
600,2588

The (x,y) points for comeatmebro's data are:
210,2299
360,2794
450,2859
540,3251
600,3265

You can enter these points in line of best fit calculators like:
http://illuminations.nctm.org/Activity.aspx?id=4186
or http://www.endmemo.com/statistics/lr.php
I used the first one.

Then you can graph them as a scatterplot and also graph the line using format y=mx + b we all learned in middle school, which is in my spreadsheet if you want to check that.

The graphs themselves are a visual check that all is as it should be, so don't see there's much room for error on the math part I checked the input several times.

Maybe show your work for how you get both as 1.8? I hope this doesn't come off as hostile I appreciate your work a lot :D
 Fenrir.Snaps
Offline
Server: Fenrir
Game: FFXI
user: Mojopojo
Posts: 1139
By Fenrir.Snaps 2017-02-15 12:22:48
Link | Quote | Reply
 
Copy/paste the data you just posted into here.

http://www.xuru.org/rt/LR.asp#CopyPaste

Both point to 1800
Offline
By Verda 2017-02-15 12:33:40
Link | Quote | Reply
 
Alright, I'll try to figure out why we disagree after work thanks :D
Offline
By Verda 2017-02-15 13:06:42
Link | Quote | Reply
 
Must've been something with typing into the calculator, I get 1.8 on both now too. Also goes to show that eyeballing verification of line of best fit doesn't really work too well >.<

I'll adjust the 1.8 in the spreadsheet, thanks.

Here's the new graph it does look more accurate:


So if we accept that 1.8 is the average noise added to each shot, then can we break that down into things like reload delay, aiming delay, and noise?
 Leviathan.Comeatmebro
Online
Server: Leviathan
Game: FFXI
user: Rairin
Posts: 6052
By Leviathan.Comeatmebro 2017-02-15 13:09:59
Link | Quote | Reply
 
Just checking in, going down the list of rapid shot tests I made earlier. Once they're all done going to organize everything into a nice zip and post, most likely sometime this evening.
 Leviathan.Comeatmebro
Online
Server: Leviathan
Game: FFXI
user: Rairin
Posts: 6052
By Leviathan.Comeatmebro 2017-02-15 14:57:30
Link | Quote | Reply
 
Small additional observation:

494/990 procs on recycle for rng99, means base recycle is almost certainly 50% without merits or any gear
 Fenrir.Snaps
Offline
Server: Fenrir
Game: FFXI
user: Mojopojo
Posts: 1139
By Fenrir.Snaps 2017-02-15 15:08:57
Link | Quote | Reply
 
re: Rapid Shot

I think the JP wiki formula works well in isolation based on my last data set. There are two ways of modeling it (empirically or using Delay)

dDelta/dRS = (base - 1.8)*.28125/100 = -0.008668770025303124 for my last data set
dDelta/dRS = (360/120)*.28125/100 = -0.0084375

The measured dDelta/dSR was -.008967808135. The empirical prediction is somewhat close to this but we seem to observe slightly better results. The full model would look like the following.

delta = (delay/120)*(1 - ss)*(1 - vs)*(1 - .28125*rs) + 1.8

You can break the 1.8 down into smaller pieces if you want to but it's largely irrelevant as it's functionally the same for modeling. That said, our data also seems to outperform this model when combining many sources of delay reduction. My guess is that there is some truncation going on between each step. If we had data for a smooth range of different values of one of these metrics we might be able to observe it in the data.
 Fenrir.Snaps
Offline
Server: Fenrir
Game: FFXI
user: Mojopojo
Posts: 1139
By Fenrir.Snaps 2017-02-15 16:12:49
Link | Quote | Reply
 
Also a side thought - I think one to approach this would be to use the general model

d = delta
D = Delay
s = Snapshot
v = Velocity Shot
r = Rapid Shot
a0, a1, a2, a3 = some constants

d = (D/a0)*(1 - s)*(1 - v)*(1 - a1*r) + a2

And regress all of our data using partial derivatives/etc.