Proposed new sysctl MIB nodes

[Jason R Thorpe]

The GCC folks have recently started using the HW_PHYSMEM and HW_USERMEM
sysctl MIB nodes to tune the behavior of the garbage collecting memory
allocator in GCC. It was pointed out there that these totally fall over
with >=4G of RAM, since it's a 32-bit quantity that returns the number
of bytes.

I'd like to propose new HW_PHYSPAGES and HW_USERPAGES MIB nodes that
return the same information, but in a 32-bit page count, instead. The
implementation is left as an exercise to the reader. I just want to get
consensus on the names, so that I can tell the GCC people about it, and
have it work on all the BSD platforms (as their current sysctl code does).

[Perry E. Metzger]

Post by Jason R Thorpe
The GCC folks have recently started using the HW_PHYSMEM and HW_USERMEM
sysctl MIB nodes to tune the behavior of the garbage collecting memory
allocator in GCC. It was pointed out there that these totally fall over
with >=4G of RAM, since it's a 32-bit quantity that returns the number
of bytes.
I'd like to propose new HW_PHYSPAGES and HW_USERPAGES MIB nodes that
return the same information, but in a 32-bit page count, instead. The
implementation is left as an exercise to the reader. I just want to get
consensus on the names, so that I can tell the GCC people about it, and
have it work on all the BSD platforms (as their current sysctl code does).

How about simply having a total memory count in quads instead? That
way we won't run out when we pass 2^48 or 49th bytes in 10 or 15
years.

Perry

---------------------------------------------------------------------
The BSD APIs Discussion Mailing List
To unsubscribe:
send "unsubscribe bsd-api-discuss" to majordomo-DZEk9q7Sfju/3pe1ocb+swC/***@public.gmane.org

[Jason R Thorpe]

Post by Perry E. Metzger

Post by Jason R Thorpe
I'd like to propose new HW_PHYSPAGES and HW_USERPAGES MIB nodes that
return the same information, but in a 32-bit page count, instead. The
implementation is left as an exercise to the reader. I just want to get
consensus on the names, so that I can tell the GCC people about it, and
have it work on all the BSD platforms (as their current sysctl code does).

How about simply having a total memory count in quads instead? That
way we won't run out when we pass 2^48 or 49th bytes in 10 or 15
years.

Ok, a u_quad (page count) it is.

[Simon Burge]

[ Sorry to chime in late ]

Post by Jason R Thorpe

Post by Perry E. Metzger

Post by Jason R Thorpe
I'd like to propose new HW_PHYSPAGES and HW_USERPAGES MIB nodes that
return the same information, but in a 32-bit page count, instead. The
implementation is left as an exercise to the reader. I just want to get
consensus on the names, so that I can tell the GCC people about it, and
have it work on all the BSD platforms (as their current sysctl code does).

How about simply having a total memory count in quads instead? That
way we won't run out when we pass 2^48 or 49th bytes in 10 or 15
years.

Ok, a u_quad (page count) it is.

A quad for page count or total bytes?

Note also that the sysctl interface is signed quads only.

Simon.
--
Simon Burge <simonb-DZEk9q7Sfju/3pe1ocb+swC/***@public.gmane.org>
NetBSD Support and Service: http://www.wasabisystems.com/

---------------------------------------------------------------------
The BSD APIs Discussion Mailing List
To unsubscribe:
send "unsubscribe bsd-api-discuss" to majordomo-DZEk9q7Sfju/3pe1ocb+swC/***@public.gmane.org

[Mike Karels]

Sorry to chip in late, but I wanted to review the thread first (including
the freebsd-arch thread).

Post by Simon Burge

Post by Jason R Thorpe
Ok, a u_quad (page count) it is.

A quad for page count or total bytes?

Although I didn't see an explicit answer (other than code, which may
have crossed the other message), I'd like to suggest that a "quad" in
bytes would be better than one in pages. Applications really don't
want the value in pages, but in bytes. Applications are highly likely
to multiply the returned value by the page size anyway, and a 64-bit
value in bytes should hold us for quite a while. It seems to me that it
would be best to return a 64-bit (or larger) value in bytes, and quad
appears to be the best way to do this. It seems to me that the only reason
to use pages is to try to squeeze the value into an int, presumably still
32 bits on many systems.

Mike

---------------------------------------------------------------------
The BSD APIs Discussion Mailing List
To unsubscribe:
send "unsubscribe bsd-api-discuss" to majordomo-DZEk9q7Sfju/3pe1ocb+swC/***@public.gmane.org

[Jason R Thorpe]

On Fri, Feb 28, 2003 at 10:54:09AM -0600, Mike Karels wrote:

Ok, I cave. Enough people want it in bytes that I don't have a problem
changing it to bytes.

The new nodes are HW_PHYSMEM64 (hw.physmem64) and HW_USERMEM64 (hw.usermem64)
then.

I will adjust the NetBSD implementation to match, and will adjust the
patch for GCC (which I haven't submitted yet).

Post by Mike Karels
Although I didn't see an explicit answer (other than code, which may
have crossed the other message), I'd like to suggest that a "quad" in
bytes would be better than one in pages. Applications really don't
want the value in pages, but in bytes. Applications are highly likely
to multiply the returned value by the page size anyway, and a 64-bit
value in bytes should hold us for quite a while. It seems to me that it
would be best to return a 64-bit (or larger) value in bytes, and quad
appears to be the best way to do this. It seems to me that the only reason
to use pages is to try to squeeze the value into an int, presumably still
32 bits on many systems.
Mike
---------------------------------------------------------------------
The BSD APIs Discussion Mailing List

[Jason R Thorpe]

Post by Perry E. Metzger

Post by Jason R Thorpe
I'd like to propose new HW_PHYSPAGES and HW_USERPAGES MIB nodes that
return the same information, but in a 32-bit page count, instead. The
implementation is left as an exercise to the reader. I just want to get
consensus on the names, so that I can tell the GCC people about it, and
have it work on all the BSD platforms (as their current sysctl code does).

How about simply having a total memory count in quads instead? That
way we won't run out when we pass 2^48 or 49th bytes in 10 or 15
years.

OK, since there hasn't been any further discussion about this, I'll take
this to mean that we're all in agreement. Here is the patch I'm checking
into NetBSD. I will check a patch into GCC to use it in a few days.

[Marcel Moolenaar]

Post by Jason R Thorpe

Post by Perry E. Metzger

Post by Jason R Thorpe
I'd like to propose new HW_PHYSPAGES and HW_USERPAGES MIB nodes that
return the same information, but in a 32-bit page count, instead. The
implementation is left as an exercise to the reader. I just want to get
consensus on the names, so that I can tell the GCC people about it, and
have it work on all the BSD platforms (as their current sysctl code does).

How about simply having a total memory count in quads instead? That
way we won't run out when we pass 2^48 or 49th bytes in 10 or 15
years.

OK, since there hasn't been any further discussion about this, I'll take
this to mean that we're all in agreement. Here is the patch I'm checking
into NetBSD. I will check a patch into GCC to use it in a few days.

Jason,

I subscribed late, so missed the discussion about this other than
the snippet Warner forwarded to arch-***@public.gmane.org It's not my
intend to trigger a rerun of the discussion, but what I'm wondering
about is what exactly the value is of knowing the amount of memory
physically installed in the machine given the fact that there are
process limits and the process is running in a multi-user, multi-
process environment?

[Jason R Thorpe]

Post by Marcel Moolenaar
I subscribed late, so missed the discussion about this other than
intend to trigger a rerun of the discussion, but what I'm wondering
about is what exactly the value is of knowing the amount of memory
physically installed in the machine given the fact that there are
process limits and the process is running in a multi-user, multi-
process environment?

GCC's garbage-collecting memory allocator uses this information,
along with rlimits (and sometimes raising rlimits) to tune itself.
The aim is to avoid making the machine thrash.

I am but the messenger ... the application is already using HW_PHYSMEM,
which loses on systems with >=4G of RAM. I just want to fix it for those
kinds of systems.

[Marcel Moolenaar]

Post by Jason R Thorpe

Post by Marcel Moolenaar
I subscribed late, so missed the discussion about this other than
intend to trigger a rerun of the discussion, but what I'm wondering
about is what exactly the value is of knowing the amount of memory
physically installed in the machine given the fact that there are
process limits and the process is running in a multi-user, multi-
process environment?

GCC's garbage-collecting memory allocator uses this information,
along with rlimits (and sometimes raising rlimits) to tune itself.
The aim is to avoid making the machine thrash.

I see. I haven't looked at the GC code, so I won't comment further.

Euh... carry on :-)

[Julian Elischer]

Since FreeBSD has had for ages:

hw.physmem: 130732032
hw.usermem: 100630528
hw.pagesize: 4096

it seems a little redundant to give teh same numbers divided by
hw.pagesize.

moving them to be defined as 64 bit ints might be good but most apps
wouldn't notice. They are ususally used in their ascii form by scripts
etc.

Post by Jason R Thorpe

Post by Perry E. Metzger

Post by Jason R Thorpe
I'd like to propose new HW_PHYSPAGES and HW_USERPAGES MIB nodes that
return the same information, but in a 32-bit page count, instead. The
implementation is left as an exercise to the reader. I just want to get
consensus on the names, so that I can tell the GCC people about it, and
have it work on all the BSD platforms (as their current sysctl code does).

How about simply having a total memory count in quads instead? That
way we won't run out when we pass 2^48 or 49th bytes in 10 or 15
years.

OK, since there hasn't been any further discussion about this, I'll take
this to mean that we're all in agreement. Here is the patch I'm checking
into NetBSD. I will check a patch into GCC to use it in a few days.

there was still some discussion on theis going on in the freeBSD arch
list last I looked.(it should have been cc'd to here but wasn't)

Post by Jason R Thorpe
--

---------------------------------------------------------------------
The BSD APIs Discussion Mailing List
To unsubscribe:
send "unsubscribe bsd-api-discuss" to majordomo-DZEk9q7Sfju/3pe1ocb+swC/***@public.gmane.org

[Jason R Thorpe]

Post by Julian Elischer
hw.physmem: 130732032
hw.usermem: 100630528
hw.pagesize: 4096
it seems a little redundant to give teh same numbers divided by
hw.pagesize.
moving them to be defined as 64 bit ints might be good but most apps
wouldn't notice. They are ususally used in their ascii form by scripts
etc.

So, please tell me what happens when you have a system with >4G of RAM.
Note, I'm assuming your physmem is returned as a 32-bit int, like NetBSD's
was.

The goal of the patch is to not divide the numbers by the page size, but
rather to provide meaningful numbers for systems with >=4G RAM. (Did you
even read my original mail??)

[Julian Elischer]

Post by Jason R Thorpe

Post by Julian Elischer
hw.physmem: 130732032
hw.usermem: 100630528
hw.pagesize: 4096
it seems a little redundant to give teh same numbers divided by
hw.pagesize.
moving them to be defined as 64 bit ints might be good but most apps
wouldn't notice. They are ususally used in their ascii form by scripts
etc.

So, please tell me what happens when you have a system with >4G of RAM.
Note, I'm assuming your physmem is returned as a 32-bit int, like NetBSD's
was.
The goal of the patch is to not divide the numbers by the page size, but
rather to provide meaningful numbers for systems with >=4G RAM. (Did you
even read my original mail??)

Yes I did..
I said I think they should just be redefined as 64 bit ints..
It's just my opinion. 64 bit page counts is ok but then
most applications I know of would have to fetch that, fetch the
pagesize and multiply them together, and use the result.
(I've mostly seen this used in scripts and they are interested in
"256MB" not "65536 pages")

Post by Jason R Thorpe
--

---------------------------------------------------------------------
The BSD APIs Discussion Mailing List
To unsubscribe:
send "unsubscribe bsd-api-discuss" to majordomo-DZEk9q7Sfju/3pe1ocb+swC/***@public.gmane.org

[Bill Sommerfeld]

Post by Jason R Thorpe
OK, since there hasn't been any further discussion about this, I'll take
this to mean that we're all in agreement. Here is the patch I'm checking
into NetBSD. I will check a patch into GCC to use it in a few days.

for better backwards-compatibility for folks using the old interfaces,
how about making them "saturating" values:

If the real value is >= 2^32, return (2^32)-1

- Bill

---------------------------------------------------------------------
The BSD APIs Discussion Mailing List
To unsubscribe:
send "unsubscribe bsd-api-discuss" to majordomo-DZEk9q7Sfju/3pe1ocb+swC/***@public.gmane.org

[Jason R Thorpe]

Post by Bill Sommerfeld

Post by Jason R Thorpe
OK, since there hasn't been any further discussion about this, I'll take
this to mean that we're all in agreement. Here is the patch I'm checking
into NetBSD. I will check a patch into GCC to use it in a few days.

for better backwards-compatibility for folks using the old interfaces,
If the real value is >= 2^32, return (2^32)-1

Yah, I'm going to fix the existing nodes to do this in NetBSD
later today.