Jump to content

Welcome to ExtremeHW

Welcome to ExtremeHW, like most online communities you must register to view or post in our community, but don't worry this is a simple free process that requires minimal information for you to signup. Be a part of ExtremeHW by signing in or creating an account.

  • Start new topics and reply to others
  • Subscribe to topics and forums to get email updates
  • Get your own profile page and make new friends
  • Send personal messages to other members.
  • Take advantage of site exclusive features.
IGNORED

New TC stats system & progress


Recommended Posts

Hi everyone,

 

@BWG asked me to check in and give you guys an update on the new stats site. Been working on it for a few weeks now and think it's in a good state to see how it works with real Folders (and not just @BWG and @Bastiaan_NL ?).

 

So for those of you who don't know, the old TC stats site was built about a decade ago, with most of the logic written in PHP. But it meant that UI, back-end and DB logic all kind of sat in the one place, making it difficult to change/maintain. I'll be honest, I didn't expect it to still be running 10 years later, so props to the guys for building/maintaining it over the years. ?

 

But since it's "the future", and we had a few new features we wanted implemented, we thought we'd try to rebuild it to make it a bit more flexible and manageable. I won't go into a lot of technical details here (though I can follow up later in another post if people are interested). The main goal was to replicate what we already had (retrieving stats for a user + passkey, creating teams of those users, comparing users within categories), while also adding some flexibility (adding multipliers based on the hardware being used, hourly updates, better distinction between front-end/back-end/DB, portability).

 

The status right now is that the following is implemented:

  • REST APIs to add hardware, TC users, and TC teams
  • Users can Fold for any Stanford team (so open to EHW, OCN and anyone else who wants to join I guess)
  • Hourly updates
  • A multiplier (based on hardware from the LARS PPD database and OS)
  • WU count in addition to points
  • Overall TC stats, and daily stats per user
  • Ranking of teams and users

 

There's no UI yet, but everything is available through a REST API in JSON. Here's a little roadmap of features I'm still working on, including:

  • 'Retiring' users from a team, while still keeping their stats for the team
  • Daily/monthly stats for teams
  • Ranking users across a category
  • Points comparison to leading team/user
  • Build a small PoC UI to make things look less... text-y for you guys
  • Look into a monitoring tool for metrics/statistics on the system
  • Documentation/architecture diagrams and some more back-end improvements, try and make this more maintainable in future

 

We're starting to run with 3 or so teams for the moment, and you can see the outputs here (not as nice as the main site, but we can make it look pretty later):
 

Stats Page

 

All the source code is available on github too, so if anyone wants to keep an eye on changes or wants to see the source, send me a PM. :)

 

//zodac

Edited by zodac
  • Thanks 4
Link to comment
Share on other sites

EHW Content Creator
1.2k 649
10 minutes ago, zodac said:

to see how it works with real Folders (and not just @BWG and @Bastiaan_NL ?).

Oh man, that one just about made me unplug my rigs. 1.6 Billion points and i'm just Pinocchio. 

 

P.S. Thank for the update ? 

 

*Edit, re-read that comment, i'm an idiot. That link is just re-directing me back to EHW

Edited by Avacado
Link to comment
Share on other sites

EHW Content Creator
1.2k 649
7 minutes ago, zodac said:

Yeah... I've edited the post like three times, just doesn't want to link to anything else. But you can copy/paste the URL as is. 

My points seem insanely high for 1 Tesla, I might have done something wrong. 

 

"18962289" It would take me a year to get 18 million points with one Tesla. 

 

That 960 is killing it @Supercrumpet 54 mil??? Yeesh. 

Edited by Avacado
Link to comment
Share on other sites

Premium Platinum
885 456
8 minutes ago, zodac said:

Yeah... I've edited the post like three times, just doesn't want to link to anything else. But you can copy/paste the URL as is. 

It's not working because of what you said about @BWG and me... :classic_smile:

 

4 minutes ago, Avacado said:

My points seem insanely high for 1 Tesla, I might have done something wrong. 

 



"18962289" It would take me a year to get 18 million points with one Tesla. 

That is after the multiplier, Tesla points on Lar systems are really low so that results in way higher "bonus" points for you!

"pointsWithoutMultiplier": 4580263 still seems like a lot, and that's with 7 units.
Edited by Bastiaan_NL
  • Thanks 1
Link to comment
Share on other sites

EHW Content Creator
1.2k 649
4 minutes ago, Bastiaan_NL said:

It's not working because of what you said about @BWG and me... :classic_smile:

 

That is after the multiplier, Tesla points on Lar systems are really low so that results in way higher "bonus" points for you!



"pointsWithoutMultiplier": 4580263 still seems like a lot..

True. Which makes me think it captured all 5? Maybe. Even so with 7 completed units, my points should be around 1.1 Mil. 

 

Have been seeing 170k per WU with the 13446 WU's. 

Edited by Avacado
Link to comment
Share on other sites

Premium Platinum
885 456
2 minutes ago, Avacado said:

True. Which makes me think it captured all 5? Maybe. Even so with 7 completed units, my points should be around 1.1 Mil. 

That was my first thought, but with 7 units 4.6 mil is a lot(read: "impossible") on Tesla's. 

 

And I've got 2.4 mil with 28 units, also a bit strange

Edited by Bastiaan_NL
Link to comment
Share on other sites

Premium Platinum
885 456
1 minute ago, zodac said:

Yeah, I've been out of the loop for a few years, so not sure what points to expect for hardware. 

 

Also, don't read into the WU count too much. That API from Stanford is a bit iffy... I'm still testing it out. 

I had the same problem a year ago, was kinda shocked that it would only take me 3 days to generate the same points as I did in my time at OCN :classic_laugh:

 

Thanks for the update.

Link to comment
Share on other sites

1 minute ago, Bastiaan_NL said:

I had the same problem a year ago, was kinda shocked that it would only take me 3 days to generate the same points as I did in my time at OCN :classic_laugh:

 

Thanks for the update.

No worries. I'll check in tomorrow and look into the points some more. We have HFM links for a few of you guys, so I'll try track unit by unit and see what happens. 

Link to comment
Share on other sites

So I think I've found the problem. I was building the query to the Stanford API incorrectly, but instead of failing it just defaults to giving the non-filtered stats for the user. Would have caught it earlier, but they deprecated the old stats API for a new one, so still getting used to it.

 

Gonna wipe everyone's stats and start again from 0 now, hope things look a bit better. Take a look in a few hours @Avacado and if the numbers make more sense. :)

  • Thanks 1
Link to comment
Share on other sites

1 hour ago, Supercrumpet said:

Just noticed this thread, I'll be keeping an eye out when things have developed again.

 

You know, good call. I'll just tag everyone in case. :)

 

FYI:

 

Foldaholics
@tictoc @axipher @Avacado

Fluffy Pink Ninja's
@Bastiaan_NL @BWG @tenchimuyo93

Out of Beta
@damric @Supercrumpet @Zonalo
 

  • Thanks 1
Link to comment
Share on other sites

2 hours ago, BWG said:

zodac just erased all memories of @axipherbeating me. :)  I like the 'wipe' feature ;) 

 

He's ahead of you right now. ?

18 minutes ago, Supercrumpet said:

Maybe I missed something but I see a couple cases of tracked WUs but no points. What's going on there?

 

Yeah, I don't think the WU count is gonna work. The old API had something we could query that gave us a clear count of WUs filtered by username/passkey/team. We got a nice response like this:
 

{
     "description": "Name is 'zodac' -- Passkey 'fc7d6837269d86784d8bfd0b386d6bca' -- Team '37726'",
     "monthly": false,
     "results": [
              {
                "wus": 22023,
                "credit_cert": "https://apps.foldingathome.org/awards?user=28431&type=score",
                "name": "zodac",
                "rank": 33481,
                "credit": 39514566,
                "team": 37726,
                "wus_cert": "https://apps.foldingathome.org/awards?user=28431&type=wus",
                "id": 28431
              }
            ],
     "month": 3,
     "year": 2021,
     "query": "donor",
     "path": "donors"
}

 

But the new API doesn't give us an endpoint like that. The closest we have is this one:
https://api2.foldingathome.org/#GET-/bonus

It has a WU count, but it's based off "bonuses", and not filtered by team. So I don't actually know what it does. And since I don't Fold at the moment, I can't really track what causes it to increase (certain WU projects, ALL WUs, etc?).

 

So I'm keeping them in for now to see how they look, and if they end up being useless we can just take them out.

Link to comment
Share on other sites

EHW Content Creator
1.2k 649
10 hours ago, zodac said:

So I think I've found the problem. I was building the query to the Stanford API incorrectly, but instead of failing it just defaults to giving the non-filtered stats for the user. Would have caught it earlier, but they deprecated the old stats API for a new one, so still getting used to it.

 

Gonna wipe everyone's stats and start again from 0 now, hope things look a bit better. Take a look in a few hours @Avacado and if the numbers make more sense. :)

Seems the new API is not registering points for me or Bastiaan. Everyone else's looks pretty good 

Link to comment
Share on other sites

2 minutes ago, Avacado said:

Seems the new API is not registering points for me or Bastiaan. Everyone else's looks pretty good 

 

Have I got the right info for you?

 

{
  "category": "nvidia_gpu",
  "displayName": "Avacado",
  "foldingTeamNumber": 239902,
  "foldingUserName": "Avacado",
  "hardwareId": 5,
  "id": 4,
  "passkey": "79f8365643c2447c79f8365643c2447c"
}

 

 

 

1 minute ago, BWG said:

I fixed your OP since you're too old to figure it out now @zodac

Yeah, age comes for us all, sadly. :D

Link to comment
Share on other sites

EHW Content Creator
1.2k 649
3 minutes ago, zodac said:

 

Have I got the right info for you?

 

{
  "category": "nvidia_gpu",
  "displayName": "Avacado",
  "foldingTeamNumber": 239902,
  "foldingUserName": "Avacado",
  "hardwareId": 5,
  "id": 4,
  "passkey": "79f8365643c2447c79f8365643c2447c"
}

 

 

 

Yeah, age comes for us all, sadly. :D

Ah crap I just realized I switched team numbers for the Fondation. That's why it's 0. Can I set individual slots to different team numbers?

Link to comment
Share on other sites

Posted (edited)
1 minute ago, Avacado said:

Ah crap I just realized I switched team numbers for the Fondation. That's why it's 0. Can I set individual slots to different team numbers?

@BWG ?

 

I can't (currently anyway) change you back and forth between team numbers, since it messes up the initial points offset.

Edited by zodac
Link to comment
Share on other sites

EHW Content Creator
1.2k 649
1 minute ago, zodac said:

@BWG ?

 

I can't (currently anyway) change you back and forth between team numbers, since it messes up the initial points offset.

No, it's fine. It just clicked. I don't mind losing the 48 hours. Don't change anything. Will test when we hop back over to EHW tomorrow.

Link to comment
Share on other sites

So, errors are kind of what we're trying to resolve with this demo run. I'm kind of glad we had trouble with@Avacado because, well, it's him hahaha. j/k, but the initial passkey pull was done on EHW and you were on OCN at the time. That just tells us the initial pull will have to be the correct team on our end.

 

We're going to table that issue for later because it's only the initial pull, and we can control that easily. But, the whole flipping teams for FAT after the initial pull, how do you see it working for that? Has that been tested? @zodac

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...

Important Information

This Website may place and access certain Cookies on your computer. ExtremeHW uses Cookies to improve your experience of using the Website and to improve our range of products and services. ExtremeHW has carefully chosen these Cookies and has taken steps to ensure that your privacy is protected and respected at all times. All Cookies used by this Website are used in accordance with current UK and EU Cookie Law. For more information please see our Privacy Policy